00000008:00010000:3.0:1789760771.674321:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760771.674344:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503731712:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.675086:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503731840:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.687508:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985626 gen 2 00000008:00010000:3.0:1789760771.687542:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760771.687547:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.687554:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.687570:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760771.687587:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503731840:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.689160:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503731968:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.699572:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985627 gen 2 00000008:00010000:3.0:1789760771.699609:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760771.699615:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.699624:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.699641:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760771.699667:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503731968:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.701733:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503732096:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.716965:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985628 gen 2 00000008:00010000:3.0:1789760771.717002:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760771.717007:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.717016:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.717076:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760771.717102:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503732096:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.718213:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503732224:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.727791:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985629 gen 2 00000008:00010000:3.0:1789760771.727828:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760771.727834:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.727842:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.727860:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760771.727885:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503732224:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.728407:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503732352:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.738368:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985630 gen 2 00000008:00010000:3.0:1789760771.738403:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760771.738409:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.738483:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.738540:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760771.738563:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503732352:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.739599:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503732480:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.754608:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985631 gen 2 00000008:00010000:3.0:1789760771.754654:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760771.754660:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.754669:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.754689:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760771.754710:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503732480:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.765664:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985632 gen 2 00000008:00010000:3.0:1789760771.765701:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760771.765707:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.765715:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.765736:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760771.765770:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503732608:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.766557:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503732736:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.774372:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985633 gen 2 00000008:00010000:3.0:1789760771.774418:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760771.774425:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.774434:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.774451:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760771.774473:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503732736:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760771.890748:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18333, 18333] 00000080:00200000:3.0:1789760771.890763:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760771.890768:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760771.890786:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75096064 i_blocks 146696 00000020:00200000:3.0:1789760771.890789:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760771.890797:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760771.890845:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75096064, count: 4096 00000080:00200000:3.0:1789760771.890852:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75096064, bytes: 4096 00000080:00200000:3.0:1789760771.890860:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760771.890867:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760771.890870:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75096064, 75100160) 0 0 00000080:00200000:3.0:1789760771.890874:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760771.890886:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18334, 18334] 00020000:00200000:3.0:1789760771.890894:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75096064 chunk: [75096064, 75100160) 75100160, 4096 00020000:00200000:3.0:1789760771.890897:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760771.890912:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18334, 18334] 00000008:00010000:3.0:1789760771.890926:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760771.890931:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760771.890941:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75096064, 75100160) append 0 append lockpos 0 00000080:00200000:3.0:1789760771.890953:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75096064=0x479e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760771.890999:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760771.891002:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75100160 00000020:00010000:3.0:1789760771.891009:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30623f8 (ffff9018d1a6ee18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760771.891013:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30623f8 (ffff9018d1a6ee18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.891016:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.891020:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30623f8 (ffff9018d1a6ee18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760771.891022:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018d1a6ee18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760771.891182:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760771.891190:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760771.891205:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760771.891283:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75096064<>75100160 00000100:00100000:3.0:1789760771.891323:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760771.891747:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503734400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.909611:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985641 gen 2 00000008:00010000:3.0:1789760771.909654:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760771.909661:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.909669:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.909686:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760771.909709:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503734400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.922994:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985642 gen 2 00000008:00010000:3.0:1789760771.923031:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760771.923065:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.923074:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.923092:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760771.923115:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503734528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.924486:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503734656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.937975:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985643 gen 2 00000008:00010000:3.0:1789760771.938012:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760771.938019:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.938247:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.938299:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760771.938322:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503734656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.939233:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503734784:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.947747:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985644 gen 2 00000008:00010000:3.0:1789760771.947791:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760771.947798:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.947806:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.947824:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760771.947848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503734784:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.963930:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985645 gen 2 00000008:00010000:3.0:1789760771.963965:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760771.963971:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.963980:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.963999:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760771.964023:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503734912:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.969865:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760771.969886:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503735040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.978923:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985646 gen 2 00000008:00010000:3.0:1789760771.978963:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760771.978969:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.978978:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.978996:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760771.979019:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503735040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.980130:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760771.980148:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503735168:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760771.995501:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985647 gen 2 00000008:00010000:3.0:1789760771.995540:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760771.995546:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.995554:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.995571:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760771.995596:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503735168:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.010593:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503735424:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.019956:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985649 gen 2 00000008:00010000:3.0:1789760772.019990:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760772.019996:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.020005:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.020022:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760772.020083:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503735424:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.021100:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503735552:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.035292:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985650 gen 2 00000008:00010000:3.0:1789760772.035339:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760772.035345:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.035354:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.035371:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760772.035394:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503735552:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.036456:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.036472:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503735680:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.047973:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985651 gen 2 00000008:00010000:3.0:1789760772.048005:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760772.048011:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.048021:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.048085:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760772.048102:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503735680:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.049339:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503735808:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.063429:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985652 gen 2 00000008:00010000:3.0:1789760772.063467:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760772.063472:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.063481:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.063500:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760772.063521:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503735808:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.066057:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.066075:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503735936:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.079140:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985653 gen 2 00000008:00010000:3.0:1789760772.079182:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760772.079189:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.079197:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.079215:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760772.079239:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503735936:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.079772:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760772.079786:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503736064:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.092731:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985654 gen 2 00000008:00010000:3.0:1789760772.092786:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760772.092793:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.092801:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.092824:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760772.092852:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503736064:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.093451:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.093475:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503736192:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.110482:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985655 gen 2 00000008:00010000:3.0:1789760772.110532:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760772.110539:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.110548:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.110573:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760772.110600:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503736192:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.111457:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760772.111471:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503736320:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.126393:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985656 gen 2 00000008:00010000:3.0:1789760772.126438:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760772.126445:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.126454:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.126686:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760772.126714:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503736320:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.134718:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503736448:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.151375:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985657 gen 2 00000008:00010000:3.0:1789760772.151413:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760772.151421:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.151429:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.151448:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760772.151479:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503736448:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.152119:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503736576:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.161064:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985658 gen 2 00000008:00010000:3.0:1789760772.161097:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760772.161104:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.161112:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.161126:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760772.161147:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503736576:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.161667:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503736704:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.177447:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985659 gen 2 00000008:00010000:3.0:1789760772.177546:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760772.177553:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.177562:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.177628:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760772.177652:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503736704:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.178340:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503736832:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.187898:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985660 gen 2 00000008:00010000:3.0:1789760772.187947:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760772.187953:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.187963:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.187981:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760772.188259:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503736832:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.188509:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503736960:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.199517:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985661 gen 2 00000008:00010000:3.0:1789760772.199553:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760772.199560:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.199569:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.199592:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760772.199616:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503736960:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760772.377606:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18367, 18367] 00000080:00200000:3.0:1789760772.377618:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.377623:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.377635:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75235328 i_blocks 146968 00000020:00200000:3.0:1789760772.377639:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.377646:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.378292:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75235328, count: 4096 00000080:00200000:3.0:1789760772.378300:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75235328, bytes: 4096 00000080:00200000:3.0:1789760772.378310:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.378317:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760772.378319:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75235328, 75239424) 0 0 00000080:00200000:3.0:1789760772.378324:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.378336:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18368, 18368] 00020000:00200000:3.0:1789760772.378343:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75235328 chunk: [75235328, 75239424) 75239424, 4096 00020000:00200000:3.0:1789760772.378347:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.378362:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18368, 18368] 00000008:00010000:3.0:1789760772.378376:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.378381:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760772.378390:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75235328, 75239424) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.378401:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75235328=0x47c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.378445:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.378448:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75239424 00000020:00010000:3.0:1789760772.378455:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30623f8 (ffff9018d1a6ee18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.378584:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30623f8 (ffff9018d1a6ee18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.378588:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.378591:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30623f8 (ffff9018d1a6ee18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.378593:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018d1a6ee18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.378600:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.378607:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.378622:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760772.380126:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75235328<>75239424 00000100:00100000:3.0:1789760772.380181:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503738752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.399181:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985675 gen 2 00000008:00010000:3.0:1789760772.399219:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760772.399225:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.399233:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.399253:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760772.399275:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503738752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.400742:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.400759:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503738880:192.168.203.158@tcp:4:dd.0 00010000:00010000:3.0:1789760772.414390:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760772.417988:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985676 gen 2 00000008:00010000:3.0:1789760772.418214:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760772.418220:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.418229:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.418379:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760772.418404:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503738880:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.418959:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503739008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.438214:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985677 gen 2 00000008:00010000:3.0:1789760772.438245:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760772.438250:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.438257:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.438301:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760772.438321:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503739008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.439068:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503739136:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.450701:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985678 gen 2 00000008:00010000:3.0:1789760772.450747:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760772.450754:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.450762:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.450788:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760772.450812:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503739136:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.452121:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503739264:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.469884:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985679 gen 2 00000100:00100000:3.0:1789760772.498604:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985681 gen 2 00000008:00010000:3.0:1789760772.498691:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760772.498697:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.498705:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.499802:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760772.499835:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503739520:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760772.544607:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18378, 18378] 00000080:00200000:3.0:1789760772.544619:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.544623:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.544637:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75280384 i_blocks 147056 00000020:00200000:3.0:1789760772.544640:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000100:00100000:3.0:1789760772.553186:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760772.553202:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503740288:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.566204:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985687 gen 2 00000008:00010000:3.0:1789760772.566256:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760772.566262:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.566271:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.566289:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760772.566321:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503740288:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760772.567076:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c300 00000008:00100000:3.0:1789760772.567169:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75288576<>75292672 00000100:00100000:3.0:1789760772.579497:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985688 gen 2 00000008:00010000:3.0:1789760772.579533:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760772.579539:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.579547:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.579608:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760772.579641:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503740416:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.580800:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503740544:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.601276:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985689 gen 2 00000008:00010000:3.0:1789760772.601314:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760772.601321:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.601330:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.601351:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760772.601377:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503740544:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.602498:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503740672:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.616529:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985690 gen 2 00000008:00010000:3.0:1789760772.616564:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760772.616570:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.616579:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.616598:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760772.616622:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503740672:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.619956:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760772.619983:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503740800:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.632754:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985691 gen 2 00000008:00010000:3.0:1789760772.632790:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760772.632797:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.632807:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.632887:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760772.632910:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503740800:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.635847:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503740928:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.646725:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985692 gen 2 00000008:00010000:3.0:1789760772.646773:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760772.646779:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.646787:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.646812:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760772.646836:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503740928:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.763554:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503741824:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.777712:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985699 gen 2 00000008:00010000:3.0:1789760772.777756:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760772.777763:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.777778:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.777795:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760772.777816:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503741824:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.778692:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.778707:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503741952:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.791570:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985700 gen 2 00000008:00010000:3.0:1789760772.791609:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760772.791615:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.791624:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.791641:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760772.791664:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503741952:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.794302:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760772.794323:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503742080:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.806535:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985701 gen 2 00000008:00010000:3.0:1789760772.806575:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760772.806582:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.806591:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.806611:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760772.806638:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503742080:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.807339:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.807353:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503742208:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.821621:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985702 gen 2 00000008:00010000:3.0:1789760772.821666:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760772.821672:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.821682:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.821704:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760772.821739:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503742208:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.823521:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760772.823546:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503742336:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.840575:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985703 gen 2 00000008:00010000:3.0:1789760772.840615:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760772.840622:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.840631:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.840650:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760772.840673:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503742336:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.841514:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.841531:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503742464:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.858897:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985704 gen 2 00000008:00010000:3.0:1789760772.858932:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760772.858938:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.858947:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.858963:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760772.858987:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503742464:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.859770:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760772.859794:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503742592:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.874168:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985705 gen 2 00000008:00010000:3.0:1789760772.874206:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760772.874213:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.874222:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.874527:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760772.874553:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503742592:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.879517:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503742720:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.889707:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985706 gen 2 00000008:00010000:3.0:1789760772.889751:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760772.889758:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.889766:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.889791:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760772.889813:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503742720:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.890475:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503742848:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.900195:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985707 gen 2 00000008:00010000:3.0:1789760772.900230:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760772.900237:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.900246:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.900263:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760772.900287:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503742848:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.900790:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503742976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.910381:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985708 gen 2 00000008:00010000:3.0:1789760772.910417:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760772.910423:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.910432:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.910448:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760772.910471:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503742976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.911355:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503743104:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.927794:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985709 gen 2 00000008:00010000:3.0:1789760772.927829:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760772.927836:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.927844:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.927861:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760772.927883:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503743104:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.928478:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.928492:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503743232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.938214:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985710 gen 2 00000080:00200000:3.0:1789760772.938629:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18403, 18403] 00000080:00200000:3.0:1789760772.938641:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.938646:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.938658:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75382784 i_blocks 147256 00000020:00200000:3.0:1789760772.938661:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.938669:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.938714:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75382784, count: 4096 00000080:00200000:3.0:1789760772.938722:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75382784, bytes: 4096 00000080:00200000:3.0:1789760772.938742:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.938749:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760772.938751:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75382784, 75386880) 0 0 00000080:00200000:3.0:1789760772.938755:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.938777:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18404, 18404] 00020000:00200000:3.0:1789760772.938785:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75382784 chunk: [75382784, 75386880) 75386880, 4096 00020000:00200000:3.0:1789760772.938788:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.938803:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18404, 18404] 00000008:00010000:3.0:1789760772.938817:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.938822:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760772.938831:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75382784, 75386880) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.938841:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75382784=0x47e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.938883:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.938885:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75386880 00000020:00010000:3.0:1789760772.938892:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.938895:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.938898:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.938901:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.938904:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.938910:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.938915:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.938928:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760772.938996:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75382784<>75386880 00000100:00100000:3.0:1789760772.939115:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760772.939130:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503743360:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.956181:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985711 gen 2 00000008:00010000:3.0:1789760772.956222:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760772.956229:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.956237:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.956251:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760772.956272:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503743360:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.966858:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985712 gen 2 00000008:00010000:3.0:1789760772.966892:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760772.966899:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.966907:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.967358:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760772.967382:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503743488:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760772.987877:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985713 gen 2 00000008:00010000:3.0:1789760772.987914:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760772.987921:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.987928:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.987953:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760772.988989:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503743616:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760773.000229:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18407, 18407] 00000080:00200000:3.0:1789760773.000241:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.000246:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.000257:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75399168 i_blocks 147288 00000020:00200000:3.0:1789760773.000261:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.000268:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.000293:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75399168, count: 4096 00000080:00200000:3.0:1789760773.000299:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75399168, bytes: 4096 00000080:00200000:3.0:1789760773.000307:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.000315:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760773.000317:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75399168, 75403264) 0 0 00000080:00200000:3.0:1789760773.000320:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.000333:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18408, 18408] 00020000:00200000:3.0:1789760773.000340:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75399168 chunk: [75399168, 75403264) 75403264, 4096 00020000:00200000:3.0:1789760773.000344:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.000360:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18408, 18408] 00000008:00010000:3.0:1789760773.000374:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.000379:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760773.000390:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75399168, 75403264) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.000402:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75399168=0x47e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.000449:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.000563:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75403264 00000020:00010000:3.0:1789760773.000571:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.000575:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.000578:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.000581:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.000583:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.000590:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.000597:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.000613:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760773.000685:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75399168<>75403264 00000100:00100000:3.0:1789760773.009146:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985715 gen 2 00000008:00010000:3.0:1789760773.009181:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760773.009187:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.009195:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.009216:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760773.009238:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503743872:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760773.166789:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18418, 18418] 00000080:00200000:3.0:1789760773.166807:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.166813:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.166830:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75444224 i_blocks 147376 00000020:00200000:3.0:1789760773.166833:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.166846:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.166886:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75444224, count: 4096 00000080:00200000:3.0:1789760773.166893:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75444224, bytes: 4096 00000080:00200000:3.0:1789760773.166903:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.166913:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760773.166915:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75444224, 75448320) 0 0 00000080:00200000:3.0:1789760773.166920:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.166937:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18419, 18419] 00020000:00200000:3.0:1789760773.166948:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75444224 chunk: [75444224, 75448320) 75448320, 4096 00020000:00200000:3.0:1789760773.166953:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.166974:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18419, 18419] 00000008:00010000:3.0:1789760773.166995:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.167002:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760773.167015:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75444224, 75448320) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.167030:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75444224=0x47f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.167112:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.167115:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75448320 00000020:00010000:3.0:1789760773.167125:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.167129:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.167132:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.167135:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.167137:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.167145:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.167152:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.167168:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760773.167280:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75444224<>75448320 00000100:00100000:3.0:1789760773.167322:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760773.167337:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503745280:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.181062:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985726 gen 2 00000008:00010000:3.0:1789760773.181115:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760773.181123:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.181135:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.181162:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760773.181191:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503745280:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760773.196764:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18420, 18420] 00000080:00200000:3.0:1789760773.196779:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.196785:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.196801:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75452416 i_blocks 147392 00000020:00200000:3.0:1789760773.196805:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.196815:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.196903:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75452416, count: 4096 00000080:00200000:3.0:1789760773.196912:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75452416, bytes: 4096 00000080:00200000:3.0:1789760773.196923:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.196931:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760773.196933:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75452416, 75456512) 0 0 00000080:00200000:3.0:1789760773.196938:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.196953:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18421, 18421] 00020000:00200000:3.0:1789760773.196961:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75452416 chunk: [75452416, 75456512) 75456512, 4096 00020000:00200000:3.0:1789760773.196966:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.196985:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18421, 18421] 00000008:00010000:3.0:1789760773.197004:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.197010:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760773.197021:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75452416, 75456512) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.197179:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75452416=0x47f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.197243:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.197246:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75456512 00000020:00010000:3.0:1789760773.197255:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.197259:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.197262:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.197266:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.197269:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.197277:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.197284:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.197301:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760773.197403:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75452416<>75456512 00000100:00100000:3.0:1789760773.211982:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985728 gen 2 00000008:00010000:3.0:1789760773.212032:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760773.212475:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.212488:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.212628:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760773.212657:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503745536:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760773.231998:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18422, 18422] 00000080:00200000:3.0:1789760773.232015:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.232022:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.232071:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75460608 i_blocks 147408 00000020:00200000:3.0:1789760773.232075:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.232086:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:3.0:1789760773.232621:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503745792:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.248755:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985730 gen 2 00000008:00010000:3.0:1789760773.248807:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760773.248960:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.248973:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.249075:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760773.249105:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503745792:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.249766:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503745920:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.267076:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985731 gen 2 00000008:00010000:3.0:1789760773.267318:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760773.267328:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.267339:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.267361:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760773.267392:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503745920:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.269470:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760773.285676:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985732 gen 2 00000008:00010000:3.0:1789760773.285722:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760773.285740:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.285751:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.285772:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760773.285800:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503746048:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.300994:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985733 gen 2 00000008:00010000:3.0:1789760773.301092:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760773.301101:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.301112:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.301136:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760773.301169:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503746176:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.302485:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503746304:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.318997:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985734 gen 2 00000008:00010000:3.0:1789760773.319072:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760773.319080:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.319091:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.319115:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760773.319147:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503746304:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.321584:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503746432:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.338833:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985735 gen 2 00000008:00010000:3.0:1789760773.339195:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760773.339204:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.339214:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.339241:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760773.339273:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503746432:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760773.455514:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18434, 18434] 00000080:00200000:3.0:1789760773.455652:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.455660:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.455678:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75509760 i_blocks 147504 00000020:00200000:3.0:1789760773.455684:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.455696:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.455754:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75509760, count: 4096 00000080:00200000:3.0:1789760773.455765:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75509760, bytes: 4096 00000080:00200000:3.0:1789760773.455780:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.456289:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760773.456292:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75509760, 75513856) 0 0 00000080:00200000:3.0:1789760773.456297:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.456317:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18435, 18435] 00020000:00200000:3.0:1789760773.456326:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75509760 chunk: [75509760, 75513856) 75513856, 4096 00020000:00200000:3.0:1789760773.456331:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.456354:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18435, 18435] 00000008:00010000:3.0:1789760773.456440:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.456448:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760773.456462:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75509760, 75513856) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.456478:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75509760=0x4803000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.456898:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.456903:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75513856 00000020:00010000:3.0:1789760773.456915:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.456919:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.456922:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.456925:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.456929:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.456941:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.456949:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.457265:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760773.457952:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75509760<>75513856 00000100:00100000:3.0:1789760773.471550:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985742 gen 2 00000008:00010000:3.0:1789760773.471588:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760773.471595:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.471603:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.471626:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760773.471652:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503747328:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.474496:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760773.474526:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503747456:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.493264:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985743 gen 2 00000008:00010000:3.0:1789760773.493305:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760773.493310:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.493318:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.493339:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760773.493363:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503747456:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.494027:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503747584:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.508632:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985744 gen 2 00000100:00100000:3.0:1789760773.642603:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503748864:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.655356:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985754 gen 2 00000008:00010000:3.0:1789760773.655405:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760773.655413:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.655424:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.655446:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760773.655478:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503748864:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.670444:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985755 gen 2 00000008:00010000:3.0:1789760773.670495:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760773.670504:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.670515:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.670541:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760773.670570:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503748992:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760773.756958:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18455, 18455] 00000080:00200000:3.0:1789760773.756970:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.756975:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.756987:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75595776 i_blocks 147672 00000020:00200000:3.0:1789760773.756990:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.756998:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.757022:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75595776, count: 4096 00000080:00200000:3.0:1789760773.757028:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75595776, bytes: 4096 00000080:00200000:3.0:1789760773.757069:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.757077:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760773.757079:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75595776, 75599872) 0 0 00000080:00200000:3.0:1789760773.757083:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.757096:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18456, 18456] 00020000:00200000:3.0:1789760773.757103:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75595776 chunk: [75595776, 75599872) 75599872, 4096 00020000:00200000:3.0:1789760773.757107:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.757121:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18456, 18456] 00000008:00010000:3.0:1789760773.757134:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.757139:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760773.757146:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75595776, 75599872) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.757156:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75595776=0x4818000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.757197:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.757199:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75599872 00000020:00010000:3.0:1789760773.757205:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.757208:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.757212:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.757214:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.757217:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.757223:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.757242:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.757257:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760773.757333:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75595776<>75599872 00000100:00100000:3.0:1789760773.757374:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503750016:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.765365:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985763 gen 2 00000008:00010000:3.0:1789760773.765401:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760773.765407:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.765414:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.765436:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760773.765469:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503750016:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760773.767225:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c900 00000008:00100000:3.0:1789760773.767373:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75599872<>75603968 00000100:00100000:3.0:1789760773.767411:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760773.767426:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503750144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.774512:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985764 gen 2 00000008:00010000:3.0:1789760773.774549:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760773.774555:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.774565:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.774584:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760773.774608:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503750144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.786909:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985765 gen 2 00000008:00010000:3.0:1789760773.786945:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760773.786953:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.786961:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.786981:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760773.787004:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503750272:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.787696:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503750400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.808630:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985766 gen 2 00000008:00010000:3.0:1789760773.808666:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760773.808672:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.808679:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.811575:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760773.811603:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503750400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.812431:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760773.812443:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503750528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.831374:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985767 gen 2 00000008:00010000:3.0:1789760773.831410:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760773.831417:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.831425:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.831443:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760773.831471:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503750528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.833588:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503750656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.844623:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985768 gen 2 00000008:00010000:3.0:1789760773.844660:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760773.844667:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.844676:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.844693:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760773.844717:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503750656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.845368:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760773.845381:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503750784:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.856877:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985769 gen 2 00000008:00010000:3.0:1789760773.856913:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760773.856919:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.856927:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.856949:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760773.856974:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503750784:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.857875:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503750912:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.868654:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985770 gen 2 00000008:00010000:3.0:1789760773.868690:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760773.868697:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.868705:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.868732:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760773.868756:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503750912:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.869999:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760773.870015:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503751040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.880213:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985771 gen 2 00000008:00010000:3.0:1789760773.880252:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760773.880259:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.880267:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.880285:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760773.880310:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503751040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.880809:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503751168:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.890237:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985772 gen 2 00000008:00010000:3.0:1789760773.890271:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760773.890276:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.890284:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.890301:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760773.890324:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503751168:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.890974:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503751296:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.898998:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985773 gen 2 00000008:00010000:3.0:1789760773.899193:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760773.899200:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.899209:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.899228:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760773.899251:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503751296:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.900006:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503751424:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.909575:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985774 gen 2 00000008:00010000:3.0:1789760773.909612:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760773.909618:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.909627:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.909645:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760773.909670:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503751424:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.910831:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503751552:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.921922:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985775 gen 2 00000008:00010000:3.0:1789760773.921960:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760773.921967:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.921975:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.921992:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760773.922015:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503751552:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760773.922484:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c540 00000008:00100000:3.0:1789760773.922641:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75649024<>75653120 00000100:00100000:3.0:1789760773.922683:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503751680:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.932595:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985776 gen 2 00000008:00010000:3.0:1789760773.932634:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760773.932641:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.932649:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.932667:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760773.932690:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503751680:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.937133:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503751808:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.944222:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985777 gen 2 00000008:00010000:3.0:1789760773.944257:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760773.944263:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.944272:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.944289:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760773.944313:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503751808:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.944840:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503751936:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.955285:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985778 gen 2 00000008:00010000:3.0:1789760773.955319:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760773.955326:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.955335:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.955354:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760773.955377:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503751936:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.955910:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503752064:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.964840:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985779 gen 2 00000008:00010000:3.0:1789760773.964876:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760773.964882:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.964891:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.964948:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760773.964973:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503752064:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.965467:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503752192:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.973665:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985780 gen 2 00000008:00010000:3.0:1789760773.973715:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760773.973733:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.973742:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.973759:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760773.973783:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503752192:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.974583:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503752320:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.984199:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985781 gen 2 00000008:00010000:3.0:1789760773.984234:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760773.984239:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.984248:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.984265:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760773.984288:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503752320:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.986914:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503752448:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.993923:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985782 gen 2 00000008:00010000:3.0:1789760773.993957:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760773.993963:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.993972:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.993991:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760773.994015:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503752448:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760773.994936:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503752576:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.003316:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985783 gen 2 00000008:00010000:3.0:1789760774.003357:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760774.003363:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.003371:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.003429:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760774.003463:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503752576:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.004288:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503752704:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.014666:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985784 gen 2 00000100:00100000:3.0:1789760774.042830:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985787 gen 2 00000008:00010000:3.0:1789760774.042860:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760774.042866:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.042873:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.042893:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760774.042913:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503753088:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760774.052498:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18481, 18481] 00000080:00200000:3.0:1789760774.052511:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.052516:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.052528:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75702272 i_blocks 147880 00000020:00200000:3.0:1789760774.052531:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.052539:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.052564:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75702272, count: 4096 00000080:00200000:3.0:1789760774.052570:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75702272, bytes: 4096 00000080:00200000:3.0:1789760774.052578:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.052584:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760774.052585:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75702272, 75706368) 0 0 00000080:00200000:3.0:1789760774.052588:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.052599:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18482, 18482] 00020000:00200000:3.0:1789760774.052605:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75702272 chunk: [75702272, 75706368) 75706368, 4096 00020000:00200000:3.0:1789760774.052608:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.052622:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18482, 18482] 00000008:00010000:3.0:1789760774.052635:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.052640:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760774.052649:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75702272, 75706368) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.052659:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75702272=0x4832000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.052703:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.052707:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75706368 00000020:00010000:3.0:1789760774.052713:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.052716:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.052719:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.052732:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.052734:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.052742:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.052747:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.052830:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760774.052904:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75702272<>75706368 00000100:00100000:3.0:1789760774.052939:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760774.052951:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503753344:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.067083:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985789 gen 2 00000008:00010000:3.0:1789760774.067183:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760774.067190:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.067199:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.067215:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760774.067238:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503753344:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.080689:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985790 gen 2 00000008:00010000:3.0:1789760774.080736:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760774.080742:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.080751:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.080773:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760774.080795:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503753472:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.081407:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760774.081421:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503753600:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.090800:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985791 gen 2 00000008:00010000:3.0:1789760774.090838:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760774.090844:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.090851:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.090871:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760774.090894:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503753600:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.091547:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503753728:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.098247:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985792 gen 2 00000008:00010000:3.0:1789760774.098282:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760774.098289:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.098298:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.098317:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760774.098341:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503753728:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.098912:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760774.098925:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503753856:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.110989:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985793 gen 2 00000008:00010000:3.0:1789760774.111024:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760774.111031:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.111186:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.111255:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760774.111280:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503753856:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.128181:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985794 gen 2 00000008:00010000:3.0:1789760774.128242:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760774.128251:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.128264:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.128292:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760774.128668:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503753984:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760774.170066:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18490, 18490] 00000080:00200000:3.0:1789760774.170085:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.170091:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.170110:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75739136 i_blocks 147952 00000020:00200000:3.0:1789760774.170113:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.170125:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:3.0:1789760774.188080:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503754624:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.204519:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985799 gen 2 00000008:00010000:3.0:1789760774.204574:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760774.204584:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.204595:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.204618:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760774.204650:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503754624:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.225389:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985800 gen 2 00000008:00010000:3.0:1789760774.225733:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760774.225742:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.225756:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.225781:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760774.225814:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503754752:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760774.240908:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18494, 18494] 00000080:00200000:3.0:1789760774.240947:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.240952:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.240965:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75755520 i_blocks 147984 00000020:00200000:3.0:1789760774.240968:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.240975:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.241011:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75755520, count: 4096 00000080:00200000:3.0:1789760774.241018:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75755520, bytes: 4096 00000080:00200000:3.0:1789760774.241028:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.241070:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760774.241072:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75755520, 75759616) 0 0 00000080:00200000:3.0:1789760774.241077:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.241090:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18495, 18495] 00020000:00200000:3.0:1789760774.241097:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75755520 chunk: [75755520, 75759616) 75759616, 4096 00020000:00200000:3.0:1789760774.241101:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.241117:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18495, 18495] 00000008:00010000:3.0:1789760774.241130:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.241135:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760774.241144:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75755520, 75759616) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.241155:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75755520=0x483f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.241194:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.241197:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75759616 00000020:00010000:3.0:1789760774.241204:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.241208:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.241211:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.241213:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.241216:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.241223:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.241227:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.257442:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18495, 18495] 00000080:00200000:3.0:1789760774.257616:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.257621:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.257633:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75759616 i_blocks 147992 00000020:00200000:3.0:1789760774.257637:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.257711:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.257872:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75759616, count: 4096 00000080:00200000:3.0:1789760774.257880:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75759616, bytes: 4096 00000080:00200000:3.0:1789760774.257889:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.257895:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760774.257897:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75759616, 75763712) 0 0 00000080:00200000:3.0:1789760774.257901:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.257915:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18496, 18496] 00020000:00200000:3.0:1789760774.257922:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75759616 chunk: [75759616, 75763712) 75763712, 4096 00020000:00200000:3.0:1789760774.257926:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.257941:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18496, 18496] 00000008:00010000:3.0:1789760774.257955:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.257961:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760774.257970:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75759616, 75763712) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.257982:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75759616=0x4840000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.258031:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.258053:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75763712 00000020:00010000:3.0:1789760774.258060:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.258064:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.258067:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.258071:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.258074:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.258081:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.258087:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.258105:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760774.258181:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75759616<>75763712 00000100:00100000:3.0:1789760774.271176:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985803 gen 2 00000008:00010000:3.0:1789760774.271213:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760774.271220:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.271229:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.271291:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760774.271317:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503755136:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760774.330675:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18500, 18500] 00000080:00200000:3.0:1789760774.330688:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.330693:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.330705:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75780096 i_blocks 148032 00000020:00200000:3.0:1789760774.330708:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.330716:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.330766:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75780096, count: 4096 00000080:00200000:3.0:1789760774.330773:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75780096, bytes: 4096 00000080:00200000:3.0:1789760774.330783:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.330790:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760774.330792:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75780096, 75784192) 0 0 00000080:00200000:3.0:1789760774.330796:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.330808:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18501, 18501] 00020000:00200000:3.0:1789760774.330815:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75780096 chunk: [75780096, 75784192) 75784192, 4096 00020000:00200000:3.0:1789760774.330819:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.330834:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18501, 18501] 00000008:00010000:3.0:1789760774.330846:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.330852:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760774.330859:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75780096, 75784192) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.330870:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75780096=0x4845000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.330913:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.330916:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75784192 00000020:00010000:3.0:1789760774.330922:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.330925:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.330928:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.330931:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.330934:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.330941:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.330946:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.330959:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760774.331030:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75780096<>75784192 00000100:00100000:3.0:1789760774.331110:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760774.331305:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503755776:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.346097:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985808 gen 2 00000008:00010000:3.0:1789760774.346136:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760774.346143:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.346151:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.346172:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760774.346194:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503755776:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760774.375307:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18503, 18503] 00000080:00200000:3.0:1789760774.375320:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.375326:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.375337:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75792384 i_blocks 148056 00000020:00200000:3.0:1789760774.375341:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.375348:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.375376:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75792384, count: 4096 00000080:00200000:3.0:1789760774.375382:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75792384, bytes: 4096 00000080:00200000:3.0:1789760774.375390:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.375398:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760774.375400:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75792384, 75796480) 0 0 00000080:00200000:3.0:1789760774.375404:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.375418:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18504, 18504] 00020000:00200000:3.0:1789760774.375425:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75792384 chunk: [75792384, 75796480) 75796480, 4096 00020000:00200000:3.0:1789760774.375428:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.375445:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18504, 18504] 00000008:00010000:3.0:1789760774.375459:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.375464:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760774.375472:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75792384, 75796480) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.375483:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75792384=0x4848000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.375528:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.375531:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75796480 00000020:00010000:3.0:1789760774.375537:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.375540:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.375543:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.375547:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.375550:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.375556:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.375562:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.375576:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760774.375649:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75792384<>75796480 00000100:00100000:3.0:1789760774.383854:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985811 gen 2 00000008:00010000:3.0:1789760774.383894:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760774.383899:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.383907:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.383929:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760774.384188:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503756160:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760774.455598:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18511, 18511] 00000080:00200000:3.0:1789760774.455610:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.455615:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.455626:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75825152 i_blocks 148120 00000020:00200000:3.0:1789760774.455629:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.455637:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.455662:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75825152, count: 4096 00000080:00200000:3.0:1789760774.455668:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75825152, bytes: 4096 00000080:00200000:3.0:1789760774.455675:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.455682:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760774.455684:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75825152, 75829248) 0 0 00000080:00200000:3.0:1789760774.455688:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.455700:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18512, 18512] 00020000:00200000:3.0:1789760774.455707:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75825152 chunk: [75825152, 75829248) 75829248, 4096 00020000:00200000:3.0:1789760774.455712:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.455737:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18512, 18512] 00000008:00010000:3.0:1789760774.455752:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.455767:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760774.455776:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75825152, 75829248) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.455787:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75825152=0x4850000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.455831:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.455834:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75829248 00000020:00010000:3.0:1789760774.455841:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.455844:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.455846:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.455850:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.455852:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.455859:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.455865:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.455879:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760774.455956:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75825152<>75829248 00000100:00100000:3.0:1789760774.471667:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985819 gen 2 00000008:00010000:3.0:1789760774.471703:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760774.471710:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.471730:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.471750:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760774.471784:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503757184:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760774.539071:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18518, 18518] 00000080:00200000:3.0:1789760774.539148:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.539154:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.539166:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75853824 i_blocks 148176 00000020:00200000:3.0:1789760774.539170:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.539299:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.539327:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75853824, count: 4096 00000080:00200000:3.0:1789760774.539333:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75853824, bytes: 4096 00000080:00200000:3.0:1789760774.539341:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.539349:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760774.539351:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75853824, 75857920) 0 0 00000080:00200000:3.0:1789760774.539355:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.539367:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18519, 18519] 00020000:00200000:3.0:1789760774.539375:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75853824 chunk: [75853824, 75857920) 75857920, 4096 00020000:00200000:3.0:1789760774.539378:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.539393:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18519, 18519] 00000008:00010000:3.0:1789760774.539407:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.539411:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760774.539420:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75853824, 75857920) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.539547:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75853824=0x4857000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.539596:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.539599:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75857920 00000020:00010000:3.0:1789760774.539607:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.539611:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.539614:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.539618:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.539620:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.539627:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.539633:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.539648:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760774.539740:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75853824<>75857920 00000100:00100000:3.0:1789760774.539780:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760774.539863:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503758080:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.548964:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985826 gen 2 00000008:00010000:3.0:1789760774.548999:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760774.549005:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.549014:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.549096:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760774.549120:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503758080:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.549852:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503758208:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.558947:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985827 gen 2 00000008:00010000:3.0:1789760774.558983:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760774.558989:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.558997:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.559017:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760774.559068:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503758208:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.560007:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503758336:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.570954:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985828 gen 2 00000008:00010000:3.0:1789760774.570991:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760774.570997:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.571005:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.571022:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760774.571076:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503758336:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.571839:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503758464:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.585529:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985829 gen 2 00000008:00010000:3.0:1789760774.585569:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760774.585576:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.585584:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.585604:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760774.585627:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503758464:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.586300:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503758592:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.597984:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985830 gen 2 00000008:00010000:3.0:1789760774.598022:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760774.598028:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.598073:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.598089:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760774.598113:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503758592:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.598780:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503758720:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.609385:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985831 gen 2 00000008:00010000:3.0:1789760774.609420:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760774.609428:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.609437:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.609465:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760774.609499:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503758720:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.610401:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503758848:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.619622:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985832 gen 2 00000008:00010000:3.0:1789760774.619658:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760774.619675:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.619684:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.619707:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760774.619740:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503758848:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760774.660331:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18528, 18528] 00000080:00200000:3.0:1789760774.660344:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.660349:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.660361:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75894784 i_blocks 148256 00000020:00200000:3.0:1789760774.660365:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.660373:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.660400:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75894784, count: 4096 00000080:00200000:3.0:1789760774.660405:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75894784, bytes: 4096 00000080:00200000:3.0:1789760774.660413:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.660420:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760774.660422:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75894784, 75898880) 0 0 00000080:00200000:3.0:1789760774.660426:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.660437:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18529, 18529] 00020000:00200000:3.0:1789760774.660443:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75894784 chunk: [75894784, 75898880) 75898880, 4096 00020000:00200000:3.0:1789760774.660447:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.660460:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18529, 18529] 00000008:00010000:3.0:1789760774.660473:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.660477:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760774.660485:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75894784, 75898880) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.660497:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75894784=0x4861000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.660539:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.660541:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75898880 00000020:00010000:3.0:1789760774.660549:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.660552:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.660555:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.660558:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.660561:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.660567:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.660573:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.660587:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760774.660660:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75894784<>75898880 00000100:00100000:3.0:1789760774.660703:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760774.660728:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503759360:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.675125:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985836 gen 2 00000008:00010000:3.0:1789760774.675161:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760774.675167:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.675189:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.675203:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760774.675228:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503759360:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760774.698008:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18530, 18530] 00000080:00200000:3.0:1789760774.698021:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.698026:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.698064:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75902976 i_blocks 148272 00000020:00200000:3.0:1789760774.698069:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.698077:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.698103:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75902976, count: 4096 00000080:00200000:3.0:1789760774.698108:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75902976, bytes: 4096 00000080:00200000:3.0:1789760774.698116:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.698124:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760774.698126:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75902976, 75907072) 0 0 00000080:00200000:3.0:1789760774.698130:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.698144:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18531, 18531] 00020000:00200000:3.0:1789760774.698151:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75902976 chunk: [75902976, 75907072) 75907072, 4096 00020000:00200000:3.0:1789760774.698155:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.698181:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18531, 18531] 00000008:00010000:3.0:1789760774.698194:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.698198:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760774.698207:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75902976, 75907072) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.698216:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75902976=0x4863000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.698258:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.698260:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75907072 00000020:00010000:3.0:1789760774.698268:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.698271:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.698274:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.698277:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.698280:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.698287:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.698292:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.698306:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760774.698383:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75902976<>75907072 00000100:00100000:3.0:1789760774.711081:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985838 gen 2 00000008:00010000:3.0:1789760774.711116:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760774.711123:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.711132:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.711143:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760774.711167:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503759616:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760774.843243:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18542, 18542] 00000080:00200000:3.0:1789760774.843261:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.843267:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.843280:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75952128 i_blocks 148368 00000020:00200000:3.0:1789760774.843285:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.843297:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.843434:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75952128, count: 4096 00000080:00200000:3.0:1789760774.843442:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75952128, bytes: 4096 00000080:00200000:3.0:1789760774.843453:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.843461:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760774.843463:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75952128, 75956224) 0 0 00000080:00200000:3.0:1789760774.843466:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.843479:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18543, 18543] 00020000:00200000:3.0:1789760774.843485:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75952128 chunk: [75952128, 75956224) 75956224, 4096 00020000:00200000:3.0:1789760774.843489:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.843505:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18543, 18543] 00000008:00010000:3.0:1789760774.843518:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.843522:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760774.843531:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75952128, 75956224) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.843542:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75952128=0x486f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.843593:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.843596:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75956224 00000020:00010000:3.0:1789760774.843603:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.843607:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.843609:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.843613:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.843616:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.843623:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.843630:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.843872:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760774.843956:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75952128<>75956224 00000100:00100000:3.0:1789760774.866510:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985850 gen 2 00000008:00010000:3.0:1789760774.866546:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760774.866552:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.866559:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.866581:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760774.866603:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503761152:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760774.929607:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18547, 18547] 00000080:00200000:3.0:1789760774.929619:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.929624:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.929637:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 75972608 i_blocks 148408 00000020:00200000:3.0:1789760774.929641:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.929649:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.929675:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 75972608, count: 4096 00000080:00200000:3.0:1789760774.929681:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 75972608, bytes: 4096 00000080:00200000:3.0:1789760774.929689:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.929696:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760774.929698:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [75972608, 75976704) 0 0 00000080:00200000:3.0:1789760774.929702:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.929726:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18548, 18548] 00020000:00200000:3.0:1789760774.929732:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 75972608 chunk: [75972608, 75976704) 75976704, 4096 00020000:00200000:3.0:1789760774.929736:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.929761:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18548, 18548] 00000008:00010000:3.0:1789760774.929775:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.929781:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760774.929792:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [75972608, 75976704) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.929803:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=75972608=0x4874000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.929848:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.929851:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 75976704 00000020:00010000:3.0:1789760774.929858:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.929861:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.929865:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.929867:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.929870:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.929887:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.929893:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.929905:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760774.929975:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 75972608<>75976704 00000100:00100000:3.0:1789760774.930013:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760774.930026:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503761792:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760774.941088:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985855 gen 2 00000008:00010000:3.0:1789760774.941123:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760774.941129:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.941137:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.941160:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760774.941183:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503761792:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760775.051206:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18556, 18556] 00000080:00200000:3.0:1789760775.051219:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.051224:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.051237:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76009472 i_blocks 148480 00000020:00200000:3.0:1789760775.051240:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.051249:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.051274:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76009472, count: 4096 00000080:00200000:3.0:1789760775.051280:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76009472, bytes: 4096 00000080:00200000:3.0:1789760775.051288:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.051295:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760775.051296:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76009472, 76013568) 0 0 00000080:00200000:3.0:1789760775.051300:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.051312:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18557, 18557] 00020000:00200000:3.0:1789760775.051319:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76009472 chunk: [76009472, 76013568) 76013568, 4096 00020000:00200000:3.0:1789760775.051323:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.051337:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18557, 18557] 00000008:00010000:3.0:1789760775.051351:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.051356:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760775.051365:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76009472, 76013568) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.051376:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76009472=0x487d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.051419:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.051423:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76013568 00000020:00010000:3.0:1789760775.051429:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.051432:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.051435:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.051439:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.051441:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.051448:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.051454:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.051468:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760775.051600:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76009472<>76013568 00000100:00100000:3.0:1789760775.063611:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985864 gen 2 00000008:00010000:3.0:1789760775.063650:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760775.063656:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.063665:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.063688:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760775.063710:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503762944:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760775.167372:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18564, 18564] 00000080:00200000:3.0:1789760775.167384:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.167389:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.167401:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76042240 i_blocks 148544 00000020:00200000:3.0:1789760775.167404:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.167412:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.167435:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76042240, count: 4096 00000080:00200000:3.0:1789760775.167452:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76042240, bytes: 4096 00000080:00200000:3.0:1789760775.167461:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.167468:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760775.167470:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76042240, 76046336) 0 0 00000080:00200000:3.0:1789760775.167474:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.167486:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18565, 18565] 00020000:00200000:3.0:1789760775.167492:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76042240 chunk: [76042240, 76046336) 76046336, 4096 00020000:00200000:3.0:1789760775.167496:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.167520:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18565, 18565] 00000008:00010000:3.0:1789760775.167533:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.167537:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760775.167545:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76042240, 76046336) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.167555:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76042240=0x4885000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.167594:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.167597:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76046336 00000020:00010000:3.0:1789760775.167603:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.167606:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.167609:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.167611:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.167614:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.167619:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.167624:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.167636:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760775.167734:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76042240<>76046336 00000100:00100000:3.0:1789760775.178774:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985872 gen 2 00000008:00010000:3.0:1789760775.178809:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760775.178815:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.178824:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.178844:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760775.178867:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503763968:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760775.275802:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18574, 18574] 00000080:00200000:3.0:1789760775.275814:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.275819:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.275830:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76083200 i_blocks 148624 00000020:00200000:3.0:1789760775.275834:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.275841:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.275865:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76083200, count: 4096 00000080:00200000:3.0:1789760775.275871:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76083200, bytes: 4096 00000080:00200000:3.0:1789760775.275880:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.275887:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760775.275893:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76083200, 76087296) 0 0 00000080:00200000:3.0:1789760775.275897:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.275909:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18575, 18575] 00020000:00200000:3.0:1789760775.275915:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76083200 chunk: [76083200, 76087296) 76087296, 4096 00020000:00200000:3.0:1789760775.275919:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.275934:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18575, 18575] 00000008:00010000:3.0:1789760775.275947:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.275952:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760775.275960:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76083200, 76087296) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.275971:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76083200=0x488f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.276012:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.276015:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76087296 00000020:00010000:3.0:1789760775.276021:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.276024:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.276027:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.276029:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.276032:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.276074:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.276081:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.276094:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760775.276162:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76083200<>76087296 00000100:00100000:3.0:1789760775.276196:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760775.276208:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503765248:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.283540:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985882 gen 2 00000008:00010000:3.0:1789760775.283579:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760775.283586:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.283595:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.283616:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760775.283650:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503765248:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760775.291189:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18576, 18576] 00000080:00200000:3.0:1789760775.291201:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.291206:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.291218:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76091392 i_blocks 148640 00000020:00200000:3.0:1789760775.291222:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.291230:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.291253:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76091392, count: 4096 00000080:00200000:3.0:1789760775.291260:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76091392, bytes: 4096 00000080:00200000:3.0:1789760775.291268:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.291274:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760775.291277:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76091392, 76095488) 0 0 00000080:00200000:3.0:1789760775.291280:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.291292:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18577, 18577] 00020000:00200000:3.0:1789760775.291298:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76091392 chunk: [76091392, 76095488) 76095488, 4096 00020000:00200000:3.0:1789760775.291301:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.291315:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18577, 18577] 00000008:00010000:3.0:1789760775.291327:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.291331:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760775.291340:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76091392, 76095488) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.291350:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76091392=0x4891000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.291384:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.291386:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76095488 00000020:00010000:3.0:1789760775.291393:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.291396:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.291400:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.291403:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.291405:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.291411:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.291417:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.299453:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18577, 18577] 00000080:00200000:3.0:1789760775.299466:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.299471:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.299483:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76095488 i_blocks 148648 00000020:00200000:3.0:1789760775.299486:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.299542:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.299569:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76095488, count: 4096 00000080:00200000:3.0:1789760775.299575:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76095488, bytes: 4096 00000080:00200000:3.0:1789760775.299582:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.299588:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760775.299590:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76095488, 76099584) 0 0 00000080:00200000:3.0:1789760775.299593:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.299606:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18578, 18578] 00020000:00200000:3.0:1789760775.299612:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76095488 chunk: [76095488, 76099584) 76099584, 4096 00020000:00200000:3.0:1789760775.299616:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.299905:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18578, 18578] 00000008:00010000:3.0:1789760775.299922:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.299927:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760775.299936:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76095488, 76099584) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.299948:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76095488=0x4892000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.299992:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.299995:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76099584 00000020:00010000:3.0:1789760775.300001:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.300004:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.300007:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.300009:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.300012:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.300019:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.300024:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.300449:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760775.300688:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76095488<>76099584 00000100:00100000:3.0:1789760775.308368:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985885 gen 2 00000008:00010000:3.0:1789760775.308404:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760775.308410:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.308418:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.308435:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760775.308457:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503765632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.309309:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503765760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.317029:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985886 gen 2 00000008:00010000:3.0:1789760775.317104:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760775.317111:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.317120:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.317138:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760775.317162:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503765760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.317833:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503765888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.323951:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985887 gen 2 00000008:00010000:3.0:1789760775.323992:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760775.323998:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.324007:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.324027:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760775.324089:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503765888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.331300:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985888 gen 2 00000008:00010000:3.0:1789760775.331333:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760775.331340:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.331349:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.331397:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760775.331419:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503766016:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760775.438492:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18591, 18591] 00000080:00200000:3.0:1789760775.438507:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.438513:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.438527:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76152832 i_blocks 148760 00000020:00200000:3.0:1789760775.438530:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.438538:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.438564:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76152832, count: 4096 00000080:00200000:3.0:1789760775.438571:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76152832, bytes: 4096 00000080:00200000:3.0:1789760775.438579:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.438586:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760775.438588:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76152832, 76156928) 0 0 00000080:00200000:3.0:1789760775.438592:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.438605:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18592, 18592] 00020000:00200000:3.0:1789760775.438612:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76152832 chunk: [76152832, 76156928) 76156928, 4096 00020000:00200000:3.0:1789760775.438616:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.438632:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18592, 18592] 00000008:00010000:3.0:1789760775.438647:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.438652:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760775.438662:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76152832, 76156928) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.438673:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76152832=0x48a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.438729:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.438732:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76156928 00000020:00010000:3.0:1789760775.438739:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.438743:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.438746:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.438756:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.438760:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.438766:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.438772:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.438786:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760775.438869:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76152832<>76156928 00000100:00100000:3.0:1789760775.446990:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985899 gen 2 00000008:00010000:3.0:1789760775.447027:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760775.447069:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.447079:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.447098:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760775.447122:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503767424:192.168.203.158@tcp:4:dd.0 00010000:00010000:3.0:1789760775.486523:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1789760775.540875:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18599, 18599] 00000080:00200000:3.0:1789760775.540890:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.540894:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.540905:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76185600 i_blocks 148824 00000020:00200000:3.0:1789760775.540908:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.540916:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.540942:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76185600, count: 4096 00000080:00200000:3.0:1789760775.540948:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76185600, bytes: 4096 00000080:00200000:3.0:1789760775.540956:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.540963:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760775.540965:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76185600, 76189696) 0 0 00000080:00200000:3.0:1789760775.540969:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.540981:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18600, 18600] 00020000:00200000:3.0:1789760775.541141:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76185600 chunk: [76185600, 76189696) 76189696, 4096 00020000:00200000:3.0:1789760775.541144:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.541161:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18600, 18600] 00000008:00010000:3.0:1789760775.541175:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.541180:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760775.541189:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76185600, 76189696) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.542254:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76185600=0x48a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.542321:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.542324:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76189696 00000020:00010000:3.0:1789760775.542332:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.542340:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.542344:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.542348:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.542351:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.542360:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.542367:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.542389:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760775.542475:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76185600<>76189696 00000100:00100000:3.0:1789760775.542522:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760775.542539:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503768448:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.550567:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985907 gen 2 00000008:00010000:3.0:1789760775.550603:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760775.550608:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.550616:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.550642:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760775.550667:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503768448:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760775.619801:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18605, 18605] 00000080:00200000:3.0:1789760775.619814:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.619819:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.619832:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76210176 i_blocks 148872 00000020:00200000:3.0:1789760775.619835:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.619843:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.619869:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76210176, count: 4096 00000080:00200000:3.0:1789760775.619875:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76210176, bytes: 4096 00000080:00200000:3.0:1789760775.619883:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.619890:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760775.619892:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76210176, 76214272) 0 0 00000080:00200000:3.0:1789760775.619897:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.619910:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18606, 18606] 00020000:00200000:3.0:1789760775.619916:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76210176 chunk: [76210176, 76214272) 76214272, 4096 00020000:00200000:3.0:1789760775.619920:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.619935:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18606, 18606] 00000008:00010000:3.0:1789760775.619950:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.619956:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760775.619965:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76210176, 76214272) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.619977:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76210176=0x48ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.620194:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.620197:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76214272 00000020:00010000:3.0:1789760775.620205:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.620208:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.620211:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.620215:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.620217:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.620224:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.620230:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.620245:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760775.620322:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76210176<>76214272 00000100:00100000:3.0:1789760775.638330:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985913 gen 2 00000008:00010000:3.0:1789760775.638371:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760775.638377:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.638385:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.638408:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760775.638431:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503769216:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760775.649934:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18607, 18607] 00000080:00200000:3.0:1789760775.649956:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.649963:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.649980:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76218368 i_blocks 148888 00000020:00200000:3.0:1789760775.649984:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.649996:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.650027:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76218368, count: 4096 00000080:00200000:3.0:1789760775.650055:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76218368, bytes: 4096 00000080:00200000:3.0:1789760775.650067:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.650076:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760775.650078:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76218368, 76222464) 0 0 00000080:00200000:3.0:1789760775.650082:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.650098:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18608, 18608] 00020000:00200000:3.0:1789760775.650107:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76218368 chunk: [76218368, 76222464) 76222464, 4096 00020000:00200000:3.0:1789760775.650112:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.650132:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18608, 18608] 00000008:00010000:3.0:1789760775.650149:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.650155:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760775.650167:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76218368, 76222464) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.650181:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76218368=0x48b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.650244:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.650247:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76222464 00000020:00010000:3.0:1789760775.650256:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.650259:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.650263:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.650267:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.650269:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.650288:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.650296:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.650315:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760775.650435:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76218368<>76222464 00000100:00100000:3.0:1789760775.650481:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760775.650502:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503769472:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.667699:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985915 gen 2 00000008:00010000:3.0:1789760775.667751:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760775.667758:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.667767:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.667838:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760775.667862:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503769472:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.668670:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503769600:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.689549:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985916 gen 2 00000008:00010000:3.0:1789760775.689586:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760775.689592:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.689600:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.689619:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760775.689642:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503769600:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.690391:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503769728:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.700814:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985917 gen 2 00000008:00010000:3.0:1789760775.700851:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760775.700858:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.700866:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.700886:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760775.700909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503769728:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.702605:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760775.702627:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503769856:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.709094:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985918 gen 2 00000008:00010000:3.0:1789760775.709126:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760775.709132:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.709140:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.709234:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760775.709257:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503769856:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.710427:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503769984:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.718415:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985919 gen 2 00000008:00010000:3.0:1789760775.718462:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760775.718468:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.718476:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.719729:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760775.719758:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503769984:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.726596:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985920 gen 2 00000008:00010000:3.0:1789760775.726634:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760775.726640:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.726649:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.726668:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760775.726691:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503770112:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760775.738086:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18614, 18614] 00000080:00200000:3.0:1789760775.738102:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.738108:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.738121:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76247040 i_blocks 148944 00000020:00200000:3.0:1789760775.738125:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.738134:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.738157:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76247040, count: 4096 00000080:00200000:3.0:1789760775.738163:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76247040, bytes: 4096 00000080:00200000:3.0:1789760775.738172:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.738180:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760775.738182:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76247040, 76251136) 0 0 00000080:00200000:3.0:1789760775.738186:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.738199:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18615, 18615] 00020000:00200000:3.0:1789760775.738206:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76247040 chunk: [76247040, 76251136) 76251136, 4096 00020000:00200000:3.0:1789760775.738210:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.738225:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18615, 18615] 00000008:00010000:3.0:1789760775.738239:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.738245:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760775.738254:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76247040, 76251136) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.738266:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76247040=0x48b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.738313:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.738316:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76251136 00000020:00010000:3.0:1789760775.738323:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.738342:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.738346:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.738349:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.738352:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.738359:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.738365:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.738380:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760775.738457:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76247040<>76251136 00000100:00100000:3.0:1789760775.738500:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760775.738518:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503770368:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.751241:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985922 gen 2 00000008:00010000:3.0:1789760775.751275:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760775.751282:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.751291:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.751311:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760775.751716:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503770368:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760775.751760:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:3.0:1789760775.751945:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76251136<>76255232 00000100:00100000:3.0:1789760775.751988:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503770496:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760775.766020:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985923 gen 2 00000008:00010000:3.0:1789760775.766105:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:3.0:1789760775.766119:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.766128:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.766137:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:3.0:1789760775.766159:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503770496:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760776.008112:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18628, 18628] 00000080:00200000:3.0:1789760776.008127:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.008133:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.008147:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76304384 i_blocks 149056 00000020:00200000:3.0:1789760776.008151:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.008162:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.008188:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76304384, count: 4096 00000080:00200000:3.0:1789760776.008195:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76304384, bytes: 4096 00000080:00200000:3.0:1789760776.008207:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.008214:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760776.008216:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76304384, 76308480) 0 0 00000080:00200000:3.0:1789760776.008220:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.008233:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18629, 18629] 00020000:00200000:3.0:1789760776.008241:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76304384 chunk: [76304384, 76308480) 76308480, 4096 00020000:00200000:3.0:1789760776.008246:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.008265:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18629, 18629] 00000008:00010000:3.0:1789760776.008280:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.008285:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760776.008295:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76304384, 76308480) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.008308:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76304384=0x48c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.008360:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.008363:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76308480 00000020:00010000:3.0:1789760776.008370:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.008373:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.008376:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.008379:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.008382:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.008389:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.008394:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.008413:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760776.010152:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76304384<>76308480 00000100:00100000:3.0:1789760776.022291:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985936 gen 2 00000008:00010000:3.0:1789760776.022326:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760776.022332:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.022340:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.022358:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760776.022381:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503772160:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.023283:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503772288:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.039923:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985937 gen 2 00000008:00010000:3.0:1789760776.039963:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760776.039969:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.039980:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.040229:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760776.040253:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503772288:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.042405:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760776.042425:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503772416:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.058194:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985938 gen 2 00000008:00010000:3.0:1789760776.058230:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760776.058237:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.058247:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.058266:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760776.058291:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503772416:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760776.059975:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cc00 00000008:00100000:3.0:1789760776.060079:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76316672<>76320768 00000100:00100000:3.0:1789760776.060115:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760776.060130:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503772544:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.069640:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985939 gen 2 00000008:00010000:3.0:1789760776.069675:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760776.069682:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.069703:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.069734:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760776.069765:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503772544:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.079557:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985940 gen 2 00000008:00010000:3.0:1789760776.079604:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760776.079612:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.079622:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.079641:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760776.079668:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503772672:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.080404:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503772800:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.092810:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985941 gen 2 00000008:00010000:3.0:1789760776.092846:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760776.092852:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.092861:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.092884:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760776.092907:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503772800:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.093768:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503772928:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.103885:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985942 gen 2 00000008:00010000:3.0:1789760776.103922:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760776.103928:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.103936:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.103956:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760776.103980:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503772928:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.104441:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503773056:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.114476:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985943 gen 2 00000008:00010000:3.0:1789760776.114520:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760776.114527:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.114536:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.114555:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760776.114580:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503773056:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.127024:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985944 gen 2 00000008:00010000:3.0:1789760776.127084:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760776.127090:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.127097:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.127115:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760776.127143:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503773184:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760776.137689:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18638, 18638] 00000080:00200000:3.0:1789760776.137701:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.137715:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.137727:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76345344 i_blocks 149136 00000020:00200000:3.0:1789760776.137730:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.137738:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.137765:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76345344, count: 4096 00000080:00200000:3.0:1789760776.137772:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76345344, bytes: 4096 00000080:00200000:3.0:1789760776.137779:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.137786:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760776.137788:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76345344, 76349440) 0 0 00000080:00200000:3.0:1789760776.137792:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.137803:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18639, 18639] 00020000:00200000:3.0:1789760776.137809:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76345344 chunk: [76345344, 76349440) 76349440, 4096 00020000:00200000:3.0:1789760776.137812:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.137826:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18639, 18639] 00000008:00010000:3.0:1789760776.137838:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.137842:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760776.137851:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76345344, 76349440) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.137862:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76345344=0x48cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.137909:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.137911:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76349440 00000020:00010000:3.0:1789760776.137918:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.137922:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.137925:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.137928:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.137931:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.137937:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.137942:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.137956:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760776.138072:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76345344<>76349440 00000100:00100000:3.0:1789760776.138115:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760776.138130:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503773440:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.150609:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985946 gen 2 00000008:00010000:3.0:1789760776.150647:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760776.150655:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.150663:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.150741:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760776.150775:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503773440:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760776.332943:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18652, 18652] 00000080:00200000:3.0:1789760776.332959:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.332965:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.332979:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76402688 i_blocks 149248 00000020:00200000:3.0:1789760776.332983:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.332993:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.333804:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76402688, count: 4096 00000080:00200000:3.0:1789760776.333815:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76402688, bytes: 4096 00000080:00200000:3.0:1789760776.333828:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.333837:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760776.333839:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76402688, 76406784) 0 0 00000080:00200000:3.0:1789760776.333843:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.333857:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18653, 18653] 00020000:00200000:3.0:1789760776.333864:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76402688 chunk: [76402688, 76406784) 76406784, 4096 00020000:00200000:3.0:1789760776.333868:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.333884:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18653, 18653] 00000008:00010000:3.0:1789760776.333909:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.333914:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760776.333923:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76402688, 76406784) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.333933:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76402688=0x48dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.333981:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.333984:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76406784 00000020:00010000:3.0:1789760776.333990:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.333993:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.333995:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a030 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.333998:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.334007:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a030 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.334014:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.334019:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.334056:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760776.337051:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76402688<>76406784 00000100:00100000:3.0:1789760776.337211:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503775232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.355324:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985960 gen 2 00000008:00010000:3.0:1789760776.355363:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760776.355369:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.355378:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.355402:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760776.355426:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503775232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.367639:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760776.367658:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503775488:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.375969:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985962 gen 2 00000008:00010000:3.0:1789760776.376005:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760776.376011:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.376020:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.376093:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760776.376117:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503775488:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.376877:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503775616:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.462962:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985969 gen 2 00000008:00010000:3.0:1789760776.463000:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760776.463079:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.463089:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.463110:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760776.463136:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503776384:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760776.523456:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18666, 18666] 00000080:00200000:3.0:1789760776.523476:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.523481:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.523494:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76460032 i_blocks 149360 00000020:00200000:3.0:1789760776.523498:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.523507:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.523532:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76460032, count: 4096 00000080:00200000:3.0:1789760776.523539:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76460032, bytes: 4096 00000080:00200000:3.0:1789760776.523549:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.523556:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760776.523559:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76460032, 76464128) 0 0 00000080:00200000:3.0:1789760776.523562:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.523574:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18667, 18667] 00020000:00200000:3.0:1789760776.523582:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76460032 chunk: [76460032, 76464128) 76464128, 4096 00020000:00200000:3.0:1789760776.523586:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.523602:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18667, 18667] 00000008:00010000:3.0:1789760776.523616:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.523622:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a030, osc lock ffff9018d2307560, flags 0x0 00000100:00100000:3.0:1789760776.526434:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503777024:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.545671:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985974 gen 2 00000008:00010000:3.0:1789760776.545719:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760776.545726:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.545734:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.545754:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760776.545777:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503777024:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.546519:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503777152:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.569838:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985975 gen 2 00000100:00100000:3.0:1789760776.613020:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985977 gen 2 00000008:00010000:3.0:1789760776.613078:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760776.613085:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.613094:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.613112:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760776.613135:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503777408:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.614636:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503777536:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.632650:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985978 gen 2 00000008:00010000:3.0:1789760776.632709:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760776.632718:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.632731:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.632957:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760776.632991:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503777536:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.635853:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760776.635885:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503777664:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.657382:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985979 gen 2 00000008:00010000:3.0:1789760776.657433:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760776.657441:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.657451:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.657474:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760776.657511:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503777664:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.679672:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985980 gen 2 00000008:00010000:3.0:1789760776.679730:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760776.679754:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.679766:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.679788:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760776.679821:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503777792:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.681529:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503777920:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.697841:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985981 gen 2 00000008:00010000:3.0:1789760776.697895:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760776.697905:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.697917:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.697982:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760776.698015:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503777920:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.700381:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503778048:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.720204:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985982 gen 2 00000008:00010000:3.0:1789760776.720257:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760776.720265:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.720278:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.720391:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760776.720425:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503778048:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.721867:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503778176:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.734210:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985983 gen 2 00000008:00010000:3.0:1789760776.734264:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760776.734273:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.734284:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.734306:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760776.734338:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503778176:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.735960:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503778304:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.750604:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985984 gen 2 00000008:00010000:3.0:1789760776.750654:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760776.750663:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.750675:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.750759:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760776.750791:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503778304:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.752395:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503778432:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.769978:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985985 gen 2 00000008:00010000:3.0:1789760776.770079:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760776.770089:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.770171:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.770191:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760776.770222:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503778432:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.771262:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503778560:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.783672:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985986 gen 2 00000008:00010000:3.0:1789760776.783735:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760776.783806:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.783819:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.783881:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760776.783913:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503778560:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.784837:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760776.784857:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503778688:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.794269:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985987 gen 2 00000008:00010000:3.0:1789760776.794323:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760776.794330:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.794341:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.794472:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760776.794513:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503778688:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.795462:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503778816:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.809204:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985988 gen 2 00000008:00010000:3.0:1789760776.809248:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760776.809255:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.809266:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.809289:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760776.809319:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503778816:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.810228:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503778944:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.829320:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985989 gen 2 00000008:00010000:3.0:1789760776.829371:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760776.829379:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.829391:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.829611:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760776.829647:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503778944:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.830352:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503779072:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.843983:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985990 gen 2 00000008:00010000:3.0:1789760776.844032:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760776.844081:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.844092:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.844112:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760776.844140:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503779072:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.846080:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503779200:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.862747:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985991 gen 2 00000008:00010000:3.0:1789760776.862806:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760776.862815:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.862826:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.862848:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760776.862880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503779200:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.863731:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760776.863751:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503779328:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.877200:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985992 gen 2 00000008:00010000:3.0:1789760776.877251:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760776.877259:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.877271:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.877295:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760776.877327:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503779328:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.878157:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503779456:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.887689:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985993 gen 2 00000008:00010000:3.0:1789760776.887799:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760776.887808:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.887819:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.887843:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760776.887874:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503779456:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.888713:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503779584:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.894614:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503779712:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760776.894998:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503779840:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760776.895095:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503780096:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760776.895157:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1789760776.895163:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503779968:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760776.895204:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503780224:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760776.899412:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503779840:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760776.899440:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503780096:192.168.203.158@tcp:400:kworker.0 00000080:00200000:3.0:1789760776.903378:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18687, 18687] 00000080:00200000:3.0:1789760776.903394:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.903400:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.903416:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76546048 i_blocks 149528 00000020:00200000:3.0:1789760776.903419:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.903429:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:3.0:1789760776.915408:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985995 gen 2 00000008:00010000:3.0:1789760776.915570:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760776.915578:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.915586:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.915603:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760776.915634:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503780352:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.916391:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503780480:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.929326:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985996 gen 2 00000008:00010000:3.0:1789760776.929433:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760776.929442:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.929452:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.929507:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760776.929537:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503780480:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.930503:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503780608:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.944614:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985997 gen 2 00000008:00010000:3.0:1789760776.944663:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760776.944672:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.944682:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.944862:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760776.944906:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503780608:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.945639:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503780736:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.958510:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985998 gen 2 00000008:00010000:3.0:1789760776.958546:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760776.958553:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.958563:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.958587:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760776.958612:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503780736:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.960604:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503780864:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.979021:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294985999 gen 2 00000080:00200000:3.0:1789760776.979634:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18692, 18692] 00000080:00200000:3.0:1789760776.979680:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.979686:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.979712:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76566528 i_blocks 149568 00000020:00200000:3.0:1789760776.979717:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.979726:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.979771:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76566528, count: 4096 00000080:00200000:3.0:1789760776.979779:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76566528, bytes: 4096 00000080:00200000:3.0:1789760776.979791:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.979801:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760776.979803:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76566528, 76570624) 0 0 00000080:00200000:3.0:1789760776.979807:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.979825:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18693, 18693] 00020000:00200000:3.0:1789760776.979834:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76566528 chunk: [76566528, 76570624) 76570624, 4096 00020000:00200000:3.0:1789760776.979838:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.979858:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18693, 18693] 00000008:00010000:3.0:1789760776.979876:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.979883:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863af30, osc lock ffff9018d2306260, flags 0x0 00000080:00200000:3.0:1789760776.979906:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76566528, 76570624) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.979921:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76566528=0x4905000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.980245:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.980248:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76570624 00000020:00010000:3.0:1789760776.980258:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.980262:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.980265:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863af30 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.980269:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418f38 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.980272:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863af30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.980280:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.980288:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.980304:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760776.980405:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76566528<>76570624 00000100:00100000:3.0:1789760776.995878:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986000 gen 2 00000008:00010000:3.0:1789760776.995940:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760776.995949:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.995960:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.995980:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760776.996011:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503780992:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760776.998637:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503781120:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.012330:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986001 gen 2 00000008:00010000:3.0:1789760777.012388:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760777.012397:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.012409:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.012435:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760777.012468:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503781120:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.029880:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503781376:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.046552:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986003 gen 2 00000008:00010000:3.0:1789760777.046605:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760777.046615:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.046625:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.046650:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760777.046682:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503781376:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.048313:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.048342:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503781504:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.059257:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986004 gen 2 00000008:00010000:3.0:1789760777.059310:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760777.059319:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.059330:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.059410:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760777.059442:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503781504:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.075017:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986005 gen 2 00000080:00200000:3.0:1789760777.075613:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18698, 18698] 00000080:00200000:3.0:1789760777.075629:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.075636:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760777.077131:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503781760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.092759:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986006 gen 2 00000008:00010000:3.0:1789760777.092819:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760777.092828:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.092840:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.092859:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760777.092894:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503781760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.093613:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760777.093635:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503781888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.113675:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986007 gen 2 00000008:00010000:3.0:1789760777.113743:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760777.113753:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.113765:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.113796:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760777.115581:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503781888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.134547:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986008 gen 2 00000008:00010000:3.0:1789760777.134602:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760777.134612:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.134624:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.134647:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760777.134679:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503782016:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.135437:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503782144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.152847:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986009 gen 2 00000008:00010000:3.0:1789760777.152906:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760777.152916:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.152928:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.152954:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760777.152986:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503782144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.153679:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.153711:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503782272:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.171221:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986010 gen 2 00000008:00010000:3.0:1789760777.171272:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760777.171281:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.171293:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.171393:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760777.171431:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503782272:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.173126:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503782400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.183305:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986011 gen 2 00000008:00010000:3.0:1789760777.183358:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760777.183367:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.183378:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.183400:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760777.183440:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503782400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.185092:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503782528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.198411:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986012 gen 2 00000008:00010000:3.0:1789760777.198467:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760777.198476:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.198488:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.198509:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760777.198541:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503782528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.199416:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760777.199437:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503782656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.218895:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986013 gen 2 00000008:00010000:3.0:1789760777.218952:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760777.218961:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.218975:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.219104:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760777.219137:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503782656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.221921:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.221950:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503782784:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.237313:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986014 gen 2 00000008:00010000:3.0:1789760777.237365:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760777.237387:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.237399:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.237423:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760777.237457:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503782784:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.239413:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760777.239442:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503782912:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.254392:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986015 gen 2 00000008:00010000:3.0:1789760777.254448:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760777.254458:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.254471:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.254495:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760777.254530:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503782912:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.257479:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.257507:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503783040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.278316:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986016 gen 2 00000008:00010000:3.0:1789760777.278371:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760777.278380:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.278392:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.278552:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760777.278582:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503783040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.286165:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760777.286192:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503783168:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.301246:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986017 gen 2 00000008:00010000:3.0:1789760777.301300:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760777.301310:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.301321:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.301348:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760777.301382:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503783168:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760777.353634:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18714, 18714] 00000080:00200000:3.0:1789760777.353650:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.353657:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.353673:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76656640 i_blocks 149744 00000020:00200000:3.0:1789760777.353676:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.353686:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.353829:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76656640, count: 4096 00000080:00200000:3.0:1789760777.353838:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76656640, bytes: 4096 00000080:00200000:3.0:1789760777.353850:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.353860:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760777.353863:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76656640, 76660736) 0 0 00000080:00200000:3.0:1789760777.353867:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.353886:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18715, 18715] 00020000:00200000:3.0:1789760777.353896:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76656640 chunk: [76656640, 76660736) 76660736, 4096 00020000:00200000:3.0:1789760777.353901:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.353922:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18715, 18715] 00000008:00010000:3.0:1789760777.353945:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.353952:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863af30, osc lock ffff9018d2306260, flags 0x0 00000080:00200000:3.0:1789760777.353964:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76656640, 76660736) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.353979:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76656640=0x491b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.354063:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.354066:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76660736 00000020:00010000:3.0:1789760777.354075:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.354078:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.354082:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863af30 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.354085:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.354088:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863af30 (ffff9018d1a6ed88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.354096:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.354103:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.354122:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760777.354520:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76656640<>76660736 00000100:00100000:3.0:1789760777.354575:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.354598:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503783808:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.366516:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986022 gen 2 00000008:00010000:3.0:1789760777.366567:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760777.366577:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.366590:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.366617:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760777.366647:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503783808:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760777.380631:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18716, 18716] 00000080:00200000:3.0:1789760777.380650:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.380655:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.380671:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76664832 i_blocks 149760 00000020:00200000:3.0:1789760777.380675:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.380686:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.380988:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76664832, count: 4096 00000080:00200000:3.0:1789760777.380997:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76664832, bytes: 4096 00000080:00200000:3.0:1789760777.381008:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.381017:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760777.381020:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76664832, 76668928) 0 0 00000080:00200000:3.0:1789760777.381024:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.381081:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18717, 18717] 00020000:00200000:3.0:1789760777.381089:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76664832 chunk: [76664832, 76668928) 76668928, 4096 00020000:00200000:3.0:1789760777.381094:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.381114:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18717, 18717] 00000008:00010000:3.0:1789760777.381132:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.381138:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863af30, osc lock ffff9018d2306260, flags 0x0 00000080:00200000:3.0:1789760777.381149:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76664832, 76668928) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.381163:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76664832=0x491d000, pages 1 (max 131072), lockless, parallel 00000100:00100000:3.0:1789760777.384721:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503784064:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.397912:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986024 gen 2 00000008:00010000:3.0:1789760777.397972:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760777.397981:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.397995:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.398152:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760777.398185:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503784064:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.400523:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503784192:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.413558:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986025 gen 2 00000008:00010000:3.0:1789760777.413597:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760777.413603:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.413610:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.413629:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760777.413652:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503784192:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.426907:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986026 gen 2 00000008:00010000:3.0:1789760777.426963:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760777.426972:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.426983:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.427009:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760777.427583:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503784320:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760777.427631:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:3.0:1789760777.427749:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76677120<>76681216 00000100:00100000:3.0:1789760777.427800:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503784448:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.438482:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986027 gen 2 00000008:00010000:3.0:1789760777.438530:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:3.0:1789760777.438539:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.438550:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.438574:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:3.0:1789760777.438607:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503784448:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760777.529429:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18726, 18726] 00000080:00200000:3.0:1789760777.529449:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.529455:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.529473:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76705792 i_blocks 149840 00000020:00200000:3.0:1789760777.529478:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.529489:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.529528:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76705792, count: 4096 00000080:00200000:3.0:1789760777.529536:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76705792, bytes: 4096 00000080:00200000:3.0:1789760777.529549:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.529558:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760777.529561:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76705792, 76709888) 0 0 00000080:00200000:3.0:1789760777.529565:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.529582:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18727, 18727] 00020000:00200000:3.0:1789760777.529591:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76705792 chunk: [76705792, 76709888) 76709888, 4096 00020000:00200000:3.0:1789760777.529596:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.529616:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18727, 18727] 00000008:00010000:3.0:1789760777.529637:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.529643:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863abb0, osc lock ffff9018d2306260, flags 0x0 00000080:00200000:3.0:1789760777.529655:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76705792, 76709888) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.529669:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76705792=0x4927000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.529741:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.529744:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76709888 00000020:00010000:3.0:1789760777.531477:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.531492:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.531499:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863abb0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.531504:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.531507:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863abb0 (ffff9018d1a6ed88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.531524:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.531536:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.531876:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760777.532017:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76705792<>76709888 00000100:00100000:3.0:1789760777.541390:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986034 gen 2 00000008:00010000:3.0:1789760777.541452:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760777.541461:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.541472:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.541505:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760777.541536:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503785344:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760777.670165:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18736, 18736] 00000080:00200000:3.0:1789760777.670188:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.670194:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.670208:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76746752 i_blocks 149920 00000020:00200000:3.0:1789760777.670213:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.670226:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.670253:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76746752, count: 4096 00000080:00200000:3.0:1789760777.670261:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76746752, bytes: 4096 00000080:00200000:3.0:1789760777.670270:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.670278:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760777.670280:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76746752, 76750848) 0 0 00000080:00200000:3.0:1789760777.670284:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.670441:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18737, 18737] 00020000:00200000:3.0:1789760777.670451:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76746752 chunk: [76746752, 76750848) 76750848, 4096 00020000:00200000:3.0:1789760777.670456:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.670477:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18737, 18737] 00000008:00010000:3.0:1789760777.670597:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.670605:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863abb0, osc lock ffff9018d2306260, flags 0x0 00000080:00200000:3.0:1789760777.670619:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76746752, 76750848) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.670632:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76746752=0x4931000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.670692:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.672256:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76750848 00000020:00010000:3.0:1789760777.672272:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.672276:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.672280:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863abb0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.672285:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.672289:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863abb0 (ffff9018d1a6ed88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.672300:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.672311:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.672344:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760777.672653:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76746752<>76750848 00000100:00100000:3.0:1789760777.672719:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.674367:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503786624:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.692769:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986044 gen 2 00000008:00010000:3.0:1789760777.692807:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760777.692814:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.692823:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.692840:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760777.692864:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503786624:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.693837:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503786752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.709811:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986045 gen 2 00000008:00010000:3.0:1789760777.709856:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760777.709866:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.709877:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.709907:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760777.709940:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503786752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.710729:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503786880:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.723438:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986046 gen 2 00000008:00010000:3.0:1789760777.723478:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760777.723486:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.723494:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.723512:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760777.723536:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503786880:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.724283:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760777.724308:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503787008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.738306:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986047 gen 2 00000008:00010000:3.0:1789760777.738342:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760777.738349:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.738357:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.738380:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760777.738404:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503787008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.740070:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503787136:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.758294:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986048 gen 2 00000100:00100000:3.0:1789760777.794242:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986051 gen 2 00000008:00010000:3.0:1789760777.794280:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760777.794286:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.794294:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.794320:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760777.794343:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503787520:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760777.877453:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18750, 18750] 00000080:00200000:3.0:1789760777.877465:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.877470:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.877482:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76804096 i_blocks 150032 00000020:00200000:3.0:1789760777.877485:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.877492:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.877518:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76804096, count: 4096 00000080:00200000:3.0:1789760777.877524:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76804096, bytes: 4096 00000080:00200000:3.0:1789760777.877531:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.877538:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760777.877540:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76804096, 76808192) 0 0 00000080:00200000:3.0:1789760777.877543:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.877556:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18751, 18751] 00020000:00200000:3.0:1789760777.877563:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76804096 chunk: [76804096, 76808192) 76808192, 4096 00020000:00200000:3.0:1789760777.877568:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.877582:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18751, 18751] 00000008:00010000:3.0:1789760777.877605:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.877610:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863abb0, osc lock ffff9018d2306260, flags 0x0 00000080:00200000:3.0:1789760777.877619:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76804096, 76808192) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.877628:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76804096=0x493f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.877671:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.877674:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76808192 00000020:00010000:3.0:1789760777.877679:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.877682:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.877684:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863abb0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.877687:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.877690:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863abb0 (ffff9018d1a6ed88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.877707:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.877713:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.877726:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760777.877811:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76804096<>76808192 00000100:00100000:3.0:1789760777.877851:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.877864:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503788416:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.889981:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986058 gen 2 00000008:00010000:3.0:1789760777.890017:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760777.890023:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.890050:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.890072:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760777.890104:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503788416:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760777.926939:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18753, 18753] 00000080:00200000:3.0:1789760777.926951:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.926956:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.926968:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 76816384 i_blocks 150056 00000020:00200000:3.0:1789760777.926971:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.926979:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.927000:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 76816384, count: 4096 00000080:00200000:3.0:1789760777.927006:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 76816384, bytes: 4096 00000080:00200000:3.0:1789760777.927017:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.927023:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760777.927025:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [76816384, 76820480) 0 0 00000080:00200000:3.0:1789760777.927029:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.927110:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18754, 18754] 00020000:00200000:3.0:1789760777.927118:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 76816384 chunk: [76816384, 76820480) 76820480, 4096 00020000:00200000:3.0:1789760777.927121:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.927137:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18754, 18754] 00000008:00010000:3.0:1789760777.927271:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.927279:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863abb0, osc lock ffff9018d2306260, flags 0x0 00000080:00200000:3.0:1789760777.927290:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [76816384, 76820480) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.927302:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=76816384=0x4942000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.927347:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.927350:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 76820480 00000020:00010000:3.0:1789760777.927356:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.927360:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.927363:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863abb0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.927366:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062a88 (ffff9018d1a6ed88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.927369:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863abb0 (ffff9018d1a6ed88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.927375:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.927380:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.927393:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760777.927853:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 76816384<>76820480 00000100:00100000:3.0:1789760777.927900:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760777.927916:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503788800:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760777.950172:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986061 gen 2 00000100:00100000:3.0:1789760778.123482:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503789952:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.133196:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986070 gen 2 00000008:00010000:3.0:1789760778.133310:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760778.133316:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.133324:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.133339:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760778.133358:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503789952:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.134091:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790080:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.145304:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986071 gen 2 00000008:00010000:3.0:1789760778.145341:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760778.145347:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.145355:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.145371:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760778.145394:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790080:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.146176:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760778.146190:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790208:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.156618:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986072 gen 2 00000008:00010000:3.0:1789760778.156652:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760778.156658:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.156666:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.156684:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760778.156718:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790208:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.159233:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790336:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.170482:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986073 gen 2 00000008:00010000:3.0:1789760778.170519:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760778.170526:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.170533:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.170553:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760778.170577:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790336:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760778.171583:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c840 00000008:00100000:3.0:1789760778.171683:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070c380 - object 0x280000401:2 offset 76869632<>76873728 00000100:00100000:3.0:1789760778.181564:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986074 gen 2 00000008:00010000:3.0:1789760778.181609:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760778.181615:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.181624:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.181643:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760778.181668:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790464:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.182247:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790592:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.194843:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986075 gen 2 00000008:00010000:3.0:1789760778.194880:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760778.194887:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.194895:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.194914:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760778.194938:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790592:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.195596:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760778.195611:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790720:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.207366:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986076 gen 2 00000008:00010000:3.0:1789760778.207402:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760778.207418:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.207428:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.207447:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760778.207472:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790720:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.208604:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760778.208623:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503790848:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.220347:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986077 gen 2 00000008:00010000:3.0:1789760778.220385:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760778.220393:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.220404:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.220424:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760778.220449:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503790848:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.222118:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760778.222140:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.234257:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986078 gen 2 00000008:00010000:3.0:1789760778.234294:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760778.234300:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.234309:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.234361:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760778.234384:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503790976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.244416:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986079 gen 2 00000008:00010000:3.0:1789760778.244461:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760778.244467:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.244475:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.244503:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760778.244526:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503791104:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.245554:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503791232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.254741:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986080 gen 2 00000008:00010000:3.0:1789760778.254776:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760778.254791:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.254800:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.254820:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760778.254846:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503791232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.256630:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760778.256648:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503791360:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.268964:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986081 gen 2 00000008:00010000:3.0:1789760778.268998:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760778.269005:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.269013:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.269032:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760778.269092:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503791360:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.270140:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760778.270157:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503791488:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.281256:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986082 gen 2 00000008:00010000:3.0:1789760778.281338:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760778.281344:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.281353:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.281408:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760778.281432:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503791488:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.282001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503791616:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.290952:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986083 gen 2 00000008:00010000:3.0:1789760778.290990:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760778.290997:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.291006:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.291025:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760778.291080:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503791616:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.291721:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503791744:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.305278:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986084 gen 2 00000008:00010000:3.0:1789760778.305314:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760778.305321:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.305331:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.305351:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760778.305376:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503791744:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.306069:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760778.306084:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503791872:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.315706:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986085 gen 2 00000008:00010000:3.0:1789760778.315744:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760778.315751:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.315759:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.315783:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760778.315807:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503791872:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.317634:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503792000:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.328902:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986086 gen 2 00000008:00010000:3.0:1789760778.328942:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760778.328949:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.328957:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.329027:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760778.329083:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503792000:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.334014:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503792128:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.352162:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986087 gen 2 00000008:00010000:3.0:1789760778.352191:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760778.352196:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.352203:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.352217:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760778.352235:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503792128:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.353230:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503792256:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.363574:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986088 gen 2 00000008:00010000:3.0:1789760778.363610:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760778.363616:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.363624:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.363645:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760778.363671:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503792256:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.364450:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503792384:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.378316:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986089 gen 2 00000008:00010000:3.0:1789760778.378352:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760778.378358:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.378366:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.378431:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760778.378456:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503792384:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.379843:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503792512:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.392728:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986090 gen 2 00000008:00010000:3.0:1789760778.392839:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760778.392846:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.392854:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.392872:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760778.392895:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503792512:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.393651:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503792640:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.409252:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986091 gen 2 00000008:00010000:3.0:1789760778.409293:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760778.409299:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.409308:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.409396:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760778.409423:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503792640:192.168.203.158@tcp:4:dd.0 00010000:00010000:3.0:1789760778.558247:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760778.606452:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503794432:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.618068:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986105 gen 2 00000008:00010000:3.0:1789760778.618103:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760778.618109:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.618119:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.618137:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760778.618162:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503794432:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.618808:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503794560:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.632620:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986106 gen 2 00000008:00010000:3.0:1789760778.632652:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760778.632658:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.632665:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.632680:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760778.632709:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503794560:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.633941:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503794688:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.647023:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986107 gen 2 00000008:00010000:3.0:1789760778.647076:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760778.647083:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.647092:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.647114:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760778.647140:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503794688:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.647917:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503794816:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.658374:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986108 gen 2 00000008:00010000:3.0:1789760778.658411:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760778.658417:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.658427:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.658445:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760778.658468:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503794816:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.661332:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503794944:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.671271:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986109 gen 2 00000008:00010000:3.0:1789760778.671308:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760778.671315:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.671322:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.671344:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760778.671372:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503794944:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.672398:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503795072:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.685448:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986110 gen 2 00000008:00010000:3.0:1789760778.685504:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760778.685511:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.685520:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.685535:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760778.685558:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503795072:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.686794:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503795200:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.697333:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986111 gen 2 00000080:00200000:3.0:1789760778.697729:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18804, 18804] 00000080:00200000:3.0:1789760778.697743:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.697749:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760778.715469:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986112 gen 2 00000080:00200000:3.0:1789760778.715919:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18805, 18805] 00000080:00200000:3.0:1789760778.715932:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.715937:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760778.716614:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503795456:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.727285:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986113 gen 2 00000008:00010000:3.0:1789760778.727317:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760778.727323:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.727330:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.727345:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760778.727365:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503795456:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.727832:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503795584:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.741335:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986114 gen 2 00000008:00010000:3.0:1789760778.741386:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760778.741393:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.741403:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.741433:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760778.741457:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503795584:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.742333:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760778.742348:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503795712:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.751713:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986115 gen 2 00000008:00010000:3.0:1789760778.751752:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760778.751759:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.751768:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.751790:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760778.751814:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503795712:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.762380:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986116 gen 2 00000008:00010000:3.0:1789760778.762416:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760778.762423:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.762432:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.762506:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760778.762531:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503795840:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.771767:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986117 gen 2 00000008:00010000:3.0:1789760778.771882:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760778.771889:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.771897:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.771917:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760778.771943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503795968:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.772485:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760778.772499:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503796096:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.779833:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986118 gen 2 00000008:00010000:3.0:1789760778.779872:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760778.779879:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.779888:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.779945:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760778.779970:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503796096:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.780665:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503796224:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.787913:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986119 gen 2 00000008:00010000:3.0:1789760778.787956:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760778.787963:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.787973:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.788113:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760778.788137:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503796224:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.788705:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503796352:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.795285:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986120 gen 2 00000008:00010000:3.0:1789760778.795321:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760778.795327:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.795336:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.795355:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760778.795379:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503796352:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.796567:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503796480:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.803924:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986121 gen 2 00000008:00010000:3.0:1789760778.803959:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760778.803965:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.803972:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.803998:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760778.804020:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503796480:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.804738:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503796608:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.813674:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986122 gen 2 00000008:00010000:3.0:1789760778.813725:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760778.813740:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.813751:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.813769:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760778.813793:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503796608:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.814376:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503796736:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.822238:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986123 gen 2 00000080:00200000:3.0:1789760778.822584:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18816, 18816] 00000080:00200000:3.0:1789760778.822624:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.822632:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.822645:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77074432 i_blocks 150560 00000020:00200000:3.0:1789760778.822648:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.822656:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:3.0:1789760778.831022:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986124 gen 2 00000008:00010000:3.0:1789760778.831092:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760778.831099:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.831108:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.831126:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760778.831148:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503796864:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.831844:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503796992:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.839607:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986125 gen 2 00000008:00010000:3.0:1789760778.839648:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760778.839654:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.839662:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.839865:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760778.839890:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503796992:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.840649:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503797120:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.848743:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986126 gen 2 00000008:00010000:3.0:1789760778.848890:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760778.848897:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.848906:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.848926:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760778.848949:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503797120:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.850315:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503797248:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.858572:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986127 gen 2 00000008:00010000:3.0:1789760778.858608:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760778.858615:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.858625:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.858697:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760778.858723:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503797248:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.859603:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503797376:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.868397:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986128 gen 2 00000080:00200000:3.0:1789760778.868739:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18821, 18821] 00000080:00200000:3.0:1789760778.868837:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.868843:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.868855:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77094912 i_blocks 150600 00000020:00200000:3.0:1789760778.868859:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.868866:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760778.868905:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77094912, count: 4096 00000080:00200000:3.0:1789760778.868912:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77094912, bytes: 4096 00000080:00200000:3.0:1789760778.868919:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760778.868927:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760778.868930:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77094912, 77099008) 0 0 00000080:00200000:3.0:1789760778.868933:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760778.868946:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18822, 18822] 00020000:00200000:3.0:1789760778.868953:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77094912 chunk: [77094912, 77099008) 77099008, 4096 00020000:00200000:3.0:1789760778.868956:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760778.868972:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18822, 18822] 00000008:00010000:3.0:1789760778.868985:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760778.868991:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863abb0, osc lock ffff9018d2306260, flags 0x0 00000080:00200000:3.0:1789760778.869000:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77094912, 77099008) append 0 append lockpos 0 00000080:00200000:3.0:1789760778.869014:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77094912=0x4986000, pages 1 (max 131072), lockless, parallel 00000100:00100000:3.0:1789760778.869458:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503797504:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.879959:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986129 gen 2 00000008:00010000:3.0:1789760778.879998:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760778.880004:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.880013:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.880069:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760778.880095:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503797504:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.880907:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503797632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.890119:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986130 gen 2 00000008:00010000:3.0:1789760778.890165:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760778.890178:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.890192:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.890219:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760778.891208:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503797632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.891713:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503797760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.900620:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986131 gen 2 00000008:00010000:3.0:1789760778.900656:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760778.900662:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.900670:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.900699:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760778.901214:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503797760:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760778.901250:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ca80 00000008:00100000:3.0:1789760778.901325:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070c380 - object 0x280000401:2 offset 77107200<>77111296 00000100:00100000:3.0:1789760778.901370:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503797888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.910246:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986132 gen 2 00000008:00010000:3.0:1789760778.910280:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760778.910286:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.910295:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.910313:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760778.910337:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503797888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.911083:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760778.911098:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503798016:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.922254:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986133 gen 2 00000008:00010000:3.0:1789760778.922295:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760778.922302:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.922311:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.922332:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760778.922356:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503798016:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.922937:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760778.922950:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503798144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.933264:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986134 gen 2 00000008:00010000:3.0:1789760778.933314:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760778.933321:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.933329:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.933352:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760778.933379:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503798144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.934142:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760778.934158:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503798272:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.943508:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986135 gen 2 00000008:00010000:3.0:1789760778.943552:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760778.943559:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.943567:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.943585:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760778.943618:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503798272:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.944239:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760778.944255:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503798400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.962779:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986136 gen 2 00000008:00010000:3.0:1789760778.962819:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760778.962827:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.962836:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.962902:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760778.962930:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503798400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.965588:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503798528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.975927:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986137 gen 2 00000008:00010000:3.0:1789760778.975975:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760778.975990:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.976000:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.976018:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760778.976061:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503798528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.976571:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503798656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760778.995632:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986138 gen 2 00000008:00010000:3.0:1789760778.995669:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760778.995675:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.995693:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.995836:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760778.995863:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503798656:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760779.009601:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18832, 18832] 00000080:00200000:3.0:1789760779.009614:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760779.009619:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760779.009632:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77139968 i_blocks 150688 00000020:00200000:3.0:1789760779.009636:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760779.009644:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760779.009670:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77139968, count: 4096 00000080:00200000:3.0:1789760779.009676:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77139968, bytes: 4096 00000080:00200000:3.0:1789760779.009693:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760779.009700:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760779.009702:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77139968, 77144064) 0 0 00000080:00200000:3.0:1789760779.009706:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760779.009717:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18833, 18833] 00020000:00200000:3.0:1789760779.009734:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77139968 chunk: [77139968, 77144064) 77144064, 4096 00020000:00200000:3.0:1789760779.009738:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760779.009753:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18833, 18833] 00000008:00010000:3.0:1789760779.009765:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760779.009769:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760779.009777:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77139968, 77144064) append 0 append lockpos 0 00000080:00200000:3.0:1789760779.009787:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77139968=0x4991000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760779.009829:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760779.009832:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77144064 00000020:00010000:3.0:1789760779.009838:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30627b8 (ffff9018d1a6e1b8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760779.009841:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30627b8 (ffff9018d1a6e1b8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.009843:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.009847:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30627b8 (ffff9018d1a6e1b8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760779.009849:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e1b8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760779.009856:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760779.009861:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760779.009875:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f13c0 00000008:00100000:3.0:1789760779.009952:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070c380 - object 0x280000401:2 offset 77139968<>77144064 00000100:00100000:3.0:1789760779.009990:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760779.010003:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503798912:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.024614:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986140 gen 2 00000008:00010000:3.0:1789760779.024651:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f13c0 00000020:00200000:3.0:1789760779.024657:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.024667:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.024702:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f13c0 00000100:00100000:3.0:1789760779.024739:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503798912:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.034258:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986141 gen 2 00000008:00010000:3.0:1789760779.034293:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760779.034300:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.034308:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.034327:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760779.034350:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503799040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.035070:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760779.035084:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503799168:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.046928:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986142 gen 2 00000008:00010000:3.0:1789760779.046964:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760779.046971:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.046980:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.047092:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760779.047117:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503799168:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.050362:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503799296:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.064276:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986143 gen 2 00000008:00010000:3.0:1789760779.064314:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760779.064321:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.064329:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.064348:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760779.064372:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503799296:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.065602:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503799424:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.078146:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986144 gen 2 00000008:00010000:3.0:1789760779.078327:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760779.078334:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.078343:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.078414:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760779.078435:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503799424:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.080991:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503799552:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.096272:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986145 gen 2 00000008:00010000:3.0:1789760779.096308:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760779.096538:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.096548:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.096608:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760779.096631:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503799552:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.097195:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503799680:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.108655:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986146 gen 2 00000008:00010000:3.0:1789760779.108826:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760779.108833:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.108843:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.108900:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760779.108926:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503799680:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.109896:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503799808:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.118664:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986147 gen 2 00000008:00010000:3.0:1789760779.118710:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760779.118716:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.118778:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.118860:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760779.118884:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503799808:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.119422:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503799936:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.128425:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986148 gen 2 00000008:00010000:3.0:1789760779.128460:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760779.128466:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.128474:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.130284:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760779.130308:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503799936:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.140858:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986149 gen 2 00000080:00200000:3.0:1789760779.141488:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18842, 18842] 00000080:00200000:3.0:1789760779.141500:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760779.141505:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760779.142562:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503800192:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.151412:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986150 gen 2 00000008:00010000:3.0:1789760779.151450:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760779.151458:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.151526:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.151846:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760779.151871:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503800192:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.152430:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503800320:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.162376:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986151 gen 2 00000008:00010000:3.0:1789760779.162424:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760779.162433:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.162443:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.162465:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760779.162491:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503800320:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.164864:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503800448:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.180932:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986152 gen 2 00000008:00010000:3.0:1789760779.180972:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760779.180979:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.180986:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.181068:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760779.181091:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503800448:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.181986:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503800576:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.192307:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986153 gen 2 00000008:00010000:3.0:1789760779.192342:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760779.192348:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.192356:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.192844:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760779.192872:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503800576:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.194537:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503800704:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.204617:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986154 gen 2 00000008:00010000:3.0:1789760779.204652:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760779.204659:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.204667:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.206381:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760779.206419:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503800704:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.224700:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986155 gen 2 00000008:00010000:3.0:1789760779.224738:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760779.224745:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.224753:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.224771:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760779.224795:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503800832:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.226638:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503800960:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.239836:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986156 gen 2 00000008:00010000:3.0:1789760779.239944:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760779.239953:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.239962:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.239982:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760779.240008:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503800960:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.240788:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503801088:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.253728:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986157 gen 2 00000008:00010000:3.0:1789760779.253774:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760779.253781:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.253789:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.254094:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760779.254116:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503801088:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.254990:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503801216:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.265079:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986158 gen 2 00000008:00010000:3.0:1789760779.265115:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760779.265121:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.265130:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.265485:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760779.265512:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503801216:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.268050:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503801344:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.285589:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986159 gen 2 00000008:00010000:3.0:1789760779.285627:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760779.285635:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.285644:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.285664:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760779.285851:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503801344:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.305980:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986160 gen 2 00000008:00010000:3.0:1789760779.306019:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760779.306025:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.306154:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.306336:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760779.306361:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503801472:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760779.358428:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18857, 18857] 00000080:00200000:3.0:1789760779.358440:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760779.358445:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760779.358457:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77242368 i_blocks 150888 00000020:00200000:3.0:1789760779.358461:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760779.358469:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760779.358516:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77242368, count: 4096 00000080:00200000:3.0:1789760779.358523:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77242368, bytes: 4096 00000080:00200000:3.0:1789760779.358532:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760779.358539:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760779.358541:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77242368, 77246464) 0 0 00000080:00200000:3.0:1789760779.358545:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760779.358558:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18858, 18858] 00020000:00200000:3.0:1789760779.358565:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77242368 chunk: [77242368, 77246464) 77246464, 4096 00020000:00200000:3.0:1789760779.358569:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760779.358584:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18858, 18858] 00000008:00010000:3.0:1789760779.358599:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760779.358603:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760779.358612:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77242368, 77246464) append 0 append lockpos 0 00000080:00200000:3.0:1789760779.358623:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77242368=0x49aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760779.358665:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760779.358667:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77246464 00000020:00010000:3.0:1789760779.358674:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760779.358678:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.358690:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.358694:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760779.358700:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760779.358707:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760779.358712:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760779.358782:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f13c0 00000008:00100000:3.0:1789760779.358854:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070c380 - object 0x280000401:2 offset 77242368<>77246464 00000100:00100000:3.0:1789760779.370364:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986165 gen 2 00000008:00010000:3.0:1789760779.370466:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f13c0 00000020:00200000:3.0:1789760779.370474:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.370485:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.370565:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f13c0 00000100:00100000:3.0:1789760779.370589:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503802112:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.554845:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503803776:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.564247:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986178 gen 2 00000008:00010000:3.0:1789760779.564282:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760779.564289:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.564298:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.564364:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760779.564388:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503803776:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.565566:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503803904:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.574190:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986179 gen 2 00000008:00010000:3.0:1789760779.574225:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760779.574231:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.574238:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.574303:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760779.574330:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503803904:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760779.679728:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18881, 18881] 00000080:00200000:3.0:1789760779.679741:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760779.679747:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760779.679763:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77340672 i_blocks 151080 00000020:00200000:3.0:1789760779.679768:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760779.679777:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760779.679803:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77340672, count: 4096 00000080:00200000:3.0:1789760779.679809:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77340672, bytes: 4096 00000080:00200000:3.0:1789760779.679817:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760779.679824:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760779.679826:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77340672, 77344768) 0 0 00000080:00200000:3.0:1789760779.679831:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760779.679843:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18882, 18882] 00020000:00200000:3.0:1789760779.679850:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77340672 chunk: [77340672, 77344768) 77344768, 4096 00020000:00200000:3.0:1789760779.679854:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760779.679868:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18882, 18882] 00000008:00010000:3.0:1789760779.679881:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760779.679886:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760779.679896:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77340672, 77344768) append 0 append lockpos 0 00000080:00200000:3.0:1789760779.679905:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77340672=0x49c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760779.679950:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760779.679953:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77344768 00000020:00010000:3.0:1789760779.679959:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760779.679963:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.679966:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.679969:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760779.679972:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760779.679978:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760779.679984:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760779.679998:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f13c0 00000008:00100000:3.0:1789760779.680675:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070c380 - object 0x280000401:2 offset 77340672<>77344768 00000100:00100000:3.0:1789760779.680840:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760779.680864:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503805184:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.709131:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503805440:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760779.747396:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18886, 18886] 00000080:00200000:3.0:1789760779.747417:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760779.747422:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760779.747435:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77361152 i_blocks 151120 00000020:00200000:3.0:1789760779.747438:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760779.747446:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760779.747474:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77361152, count: 4096 00000080:00200000:3.0:1789760779.747480:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77361152, bytes: 4096 00000080:00200000:3.0:1789760779.747488:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760779.747504:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760779.747506:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77361152, 77365248) 0 0 00000080:00200000:3.0:1789760779.747510:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760779.747522:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18887, 18887] 00020000:00200000:3.0:1789760779.747529:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77361152 chunk: [77361152, 77365248) 77365248, 4096 00020000:00200000:3.0:1789760779.747533:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760779.747548:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18887, 18887] 00000008:00010000:3.0:1789760779.747562:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760779.747568:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760779.747577:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77361152, 77365248) append 0 append lockpos 0 00000080:00200000:3.0:1789760779.747588:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77361152=0x49c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760779.747796:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760779.747799:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77365248 00000020:00010000:3.0:1789760779.747807:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760779.747810:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.747813:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.747815:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760779.747818:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760779.747825:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760779.747830:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760779.747845:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1d80 00000008:00100000:3.0:1789760779.747928:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070e680 - object 0x280000401:2 offset 77361152<>77365248 00000100:00100000:3.0:1789760779.747965:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760779.747980:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503805824:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760779.762145:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18887, 18887] 00000080:00200000:3.0:1789760779.762162:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760779.762167:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760779.762181:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77365248 i_blocks 151128 00000020:00200000:3.0:1789760779.762185:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760779.762197:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760779.762220:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77365248, count: 4096 00000080:00200000:3.0:1789760779.762226:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77365248, bytes: 4096 00000080:00200000:3.0:1789760779.762323:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760779.762333:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760779.762335:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77365248, 77369344) 0 0 00000080:00200000:3.0:1789760779.762339:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760779.762353:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18888, 18888] 00020000:00200000:3.0:1789760779.762360:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77365248 chunk: [77365248, 77369344) 77369344, 4096 00020000:00200000:3.0:1789760779.762364:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760779.762380:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18888, 18888] 00000008:00010000:3.0:1789760779.762394:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760779.762398:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760779.762407:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77365248, 77369344) append 0 append lockpos 0 00000080:00200000:3.0:1789760779.762418:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77365248=0x49c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760779.762467:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760779.762469:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77369344 00000020:00010000:3.0:1789760779.762476:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760779.762478:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.762481:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.762484:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760779.762487:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760779.762494:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760779.762505:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760779.762588:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1180 00000008:00100000:3.0:1789760779.762669:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070e300 - object 0x280000401:2 offset 77365248<>77369344 00000080:00200000:3.0:1789760779.777006:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18888, 18888] 00000080:00200000:3.0:1789760779.777020:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760779.777026:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760779.777059:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77369344 i_blocks 151136 00000020:00200000:3.0:1789760779.777063:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760779.777072:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760779.777095:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77369344, count: 4096 00000080:00200000:3.0:1789760779.777102:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77369344, bytes: 4096 00000080:00200000:3.0:1789760779.777110:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760779.777117:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760779.777119:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77369344, 77373440) 0 0 00000080:00200000:3.0:1789760779.777122:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760779.777134:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18889, 18889] 00020000:00200000:3.0:1789760779.777141:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77369344 chunk: [77369344, 77373440) 77373440, 4096 00020000:00200000:3.0:1789760779.777144:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760779.777157:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18889, 18889] 00000008:00010000:3.0:1789760779.777170:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760779.777174:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760779.777183:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77369344, 77373440) append 0 append lockpos 0 00000080:00200000:3.0:1789760779.777193:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77369344=0x49c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760779.777236:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760779.777239:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77373440 00000020:00010000:3.0:1789760779.777245:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760779.777248:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.777251:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.777254:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062d58 (ffff9018d1a6e008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760779.777257:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760779.777263:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760779.777269:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760779.777282:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760779.777358:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 77369344<>77373440 00000100:00100000:3.0:1789760779.788476:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986196 gen 2 00000008:00010000:3.0:1789760779.788524:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760779.788531:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.788540:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.788561:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760779.788586:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503806080:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760779.880266:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18895, 18895] 00000080:00200000:3.0:1789760779.880278:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760779.880283:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760779.880854:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503806976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.891298:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986203 gen 2 00000008:00010000:3.0:1789760779.891333:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760779.891340:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.891348:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.891368:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760779.891389:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503806976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.891844:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503807104:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.902651:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986204 gen 2 00000008:00010000:3.0:1789760779.902905:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760779.902912:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.902922:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.902936:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760779.902961:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503807104:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.903375:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760779.903388:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503807232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760779.918205:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986205 gen 2 00000100:00100000:3.0:1789760780.035029:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503808512:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.043355:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986215 gen 2 00000008:00010000:3.0:1789760780.043392:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760780.043398:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.043696:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.043722:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760780.044131:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503808512:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.045510:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503808640:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.059343:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986216 gen 2 00000008:00010000:3.0:1789760780.059381:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760780.059388:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.059397:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.059533:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760780.059560:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503808640:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760780.084905:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18910, 18910] 00000080:00200000:3.0:1789760780.084922:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.084927:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.084940:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77459456 i_blocks 151312 00000020:00200000:3.0:1789760780.084944:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.084955:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760780.085284:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77459456, count: 4096 00000080:00200000:3.0:1789760780.085295:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77459456, bytes: 4096 00000080:00200000:3.0:1789760780.085309:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760780.085317:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760780.085319:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77459456, 77463552) 0 0 00000080:00200000:3.0:1789760780.085323:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760780.085336:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18911, 18911] 00020000:00200000:3.0:1789760780.085343:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77459456 chunk: [77459456, 77463552) 77463552, 4096 00020000:00200000:3.0:1789760780.085346:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760780.085362:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18911, 18911] 00000008:00010000:3.0:1789760780.085376:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760780.085381:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760780.085390:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77459456, 77463552) append 0 append lockpos 0 00000080:00200000:3.0:1789760780.085401:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77459456=0x49df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760780.085527:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760780.085530:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77463552 00000020:00010000:3.0:1789760780.085538:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760780.085542:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.085544:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.085547:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760780.085550:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760780.085556:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760780.085562:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760780.085586:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760780.085669:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 77459456<>77463552 00000100:00100000:3.0:1789760780.099446:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986218 gen 2 00000008:00010000:3.0:1789760780.099482:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760780.099489:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.099498:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.099519:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760780.099543:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503808896:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760780.112624:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18912, 18912] 00000080:00200000:3.0:1789760780.112636:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.112642:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.112654:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77467648 i_blocks 151328 00000020:00200000:3.0:1789760780.112657:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.112666:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:3.0:1789760780.114981:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503809152:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.127165:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986220 gen 2 00000008:00010000:3.0:1789760780.127212:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760780.127218:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.127227:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.127291:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760780.127316:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503809152:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.128360:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503809280:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.142503:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986221 gen 2 00000100:00100000:3.0:1789760780.206702:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986225 gen 2 00000008:00010000:3.0:1789760780.206745:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760780.206752:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.206761:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.206890:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760780.206916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503809792:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.207596:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503809920:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.219950:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986226 gen 2 00000008:00010000:3.0:1789760780.220208:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760780.220215:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.220224:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.220242:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760780.220265:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503809920:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.231019:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986227 gen 2 00000008:00010000:3.0:1789760780.231184:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760780.231190:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.231199:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.231258:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760780.231281:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503810048:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.234413:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503810176:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.246889:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986228 gen 2 00000008:00010000:3.0:1789760780.246924:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760780.246931:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.246939:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.246959:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760780.246985:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503810176:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.257030:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986229 gen 2 00000100:00100000:3.0:1789760780.311016:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986232 gen 2 00000008:00010000:3.0:1789760780.311268:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760780.311275:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.311284:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.311344:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760780.311367:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503810688:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.316089:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503810816:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.329527:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986233 gen 2 00000008:00010000:3.0:1789760780.329564:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760780.329580:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.329589:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.329608:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760780.329632:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503810816:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.345483:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986234 gen 2 00000008:00010000:3.0:1789760780.345587:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760780.345593:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.345603:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.345666:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760780.345699:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503810944:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.348768:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503811072:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.369113:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986235 gen 2 00000008:00010000:3.0:1789760780.369153:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760780.369161:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.369181:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.369202:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760780.369229:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503811072:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.369952:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503811200:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.385421:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986236 gen 2 00000008:00010000:3.0:1789760780.385457:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760780.385463:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.385472:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.385500:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760780.385526:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503811200:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.387524:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760780.387548:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503811328:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.403621:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986237 gen 2 00000008:00010000:3.0:1789760780.403660:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760780.403666:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.403689:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.403708:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760780.405122:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503811328:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.419376:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986238 gen 2 00000008:00010000:3.0:1789760780.419423:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760780.419430:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.419439:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.419457:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760780.419480:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503811456:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.420335:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503811584:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.432733:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986239 gen 2 00000008:00010000:3.0:1789760780.432768:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760780.432774:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.432781:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.432798:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760780.432822:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503811584:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.433901:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760780.433919:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503811712:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.445600:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986240 gen 2 00000008:00010000:3.0:1789760780.445636:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760780.445642:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.445651:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.445671:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760780.445706:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503811712:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760780.446178:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000008:00100000:3.0:1789760780.446251:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 77553664<>77557760 00000100:00100000:3.0:1789760780.446294:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503811840:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.457134:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986241 gen 2 00000008:00010000:3.0:1789760780.457612:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760780.457619:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.457627:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.457649:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760780.457684:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503811840:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.458837:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503811968:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.465581:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986242 gen 2 00000008:00010000:3.0:1789760780.465618:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760780.465624:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.465633:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.465652:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760780.465684:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503811968:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.478520:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986243 gen 2 00000008:00010000:3.0:1789760780.478554:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760780.478560:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.478568:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.478587:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760780.478612:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503812096:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760780.585772:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18945, 18945] 00000080:00200000:3.0:1789760780.585805:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.585811:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.585822:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77602816 i_blocks 151592 00000020:00200000:3.0:1789760780.585825:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.585832:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760780.585869:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77602816, count: 4096 00000080:00200000:3.0:1789760780.585876:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77602816, bytes: 4096 00000080:00200000:3.0:1789760780.585885:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760780.585893:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760780.585895:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77602816, 77606912) 0 0 00000080:00200000:3.0:1789760780.585899:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760780.585913:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18946, 18946] 00020000:00200000:3.0:1789760780.585920:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77602816 chunk: [77602816, 77606912) 77606912, 4096 00020000:00200000:3.0:1789760780.585924:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760780.585940:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18946, 18946] 00000008:00010000:3.0:1789760780.585953:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760780.585958:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760780.585967:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77602816, 77606912) append 0 append lockpos 0 00000080:00200000:3.0:1789760780.585978:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77602816=0x4a02000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760780.586024:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760780.586027:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77606912 00000020:00010000:3.0:1789760780.588519:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760780.588531:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.588536:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.588541:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760780.588544:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760780.588556:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760780.588564:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760780.588593:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760780.588692:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 77602816<>77606912 00000100:00100000:3.0:1789760780.588752:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503813376:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.602004:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986253 gen 2 00000008:00010000:3.0:1789760780.602485:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760780.602624:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.602633:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.602721:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760780.602748:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503813376:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760780.639987:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18949, 18949] 00000080:00200000:3.0:1789760780.639999:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.640005:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.640017:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77619200 i_blocks 151624 00000020:00200000:3.0:1789760780.640020:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.640027:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760780.640095:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77619200, count: 4096 00000080:00200000:3.0:1789760780.640102:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77619200, bytes: 4096 00000080:00200000:3.0:1789760780.640109:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760780.640116:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760780.640118:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77619200, 77623296) 0 0 00000080:00200000:3.0:1789760780.640121:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760780.640133:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18950, 18950] 00020000:00200000:3.0:1789760780.640139:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77619200 chunk: [77619200, 77623296) 77623296, 4096 00020000:00200000:3.0:1789760780.640142:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760780.640156:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18950, 18950] 00000008:00010000:3.0:1789760780.640170:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760780.640175:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760780.640184:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77619200, 77623296) append 0 append lockpos 0 00000080:00200000:3.0:1789760780.640194:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77619200=0x4a06000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760780.640238:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760780.640240:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77623296 00000020:00010000:3.0:1789760780.640248:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760780.640251:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.640253:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.640257:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760780.640260:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760780.640266:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760780.640271:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760780.640284:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760780.640362:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 77619200<>77623296 00000100:00100000:3.0:1789760780.640411:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760780.640425:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503813888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.647242:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986257 gen 2 00000008:00010000:3.0:1789760780.647277:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760780.647283:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.647290:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.647365:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760780.647389:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503813888:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760780.684008:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18953, 18953] 00000080:00200000:3.0:1789760780.684024:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.684030:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.684070:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77635584 i_blocks 151656 00000020:00200000:3.0:1789760780.684075:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.684085:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:3.0:1789760780.685557:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503814400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.697615:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986261 gen 2 00000008:00010000:3.0:1789760780.697654:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760780.697661:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.697670:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.697696:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760780.698599:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503814400:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760780.698633:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000008:00100000:3.0:1789760780.699024:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 77639680<>77643776 00000100:00100000:3.0:1789760780.699064:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760780.699076:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503814528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.713666:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986262 gen 2 00000008:00010000:3.0:1789760780.713707:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760780.713801:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.713811:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.713827:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760780.713848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503814528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.724862:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986263 gen 2 00000008:00010000:3.0:1789760780.724912:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760780.724919:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.724928:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.724945:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760780.724985:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503814656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.725796:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503814784:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.736563:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986264 gen 2 00000008:00010000:3.0:1789760780.736598:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760780.736604:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.736612:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.736633:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760780.736657:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503814784:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.738166:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760780.738179:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503814912:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.745765:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986265 gen 2 00000008:00010000:3.0:1789760780.745802:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760780.745808:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.745817:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.745839:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760780.745865:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503814912:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.746782:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503815040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.762647:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986266 gen 2 00000008:00010000:3.0:1789760780.762693:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760780.762700:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.762875:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.762936:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760780.762962:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503815040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.764381:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503815168:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.777462:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986267 gen 2 00000008:00010000:3.0:1789760780.777499:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760780.777506:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.777515:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.777535:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760780.777557:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503815168:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.778419:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503815296:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.793268:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986268 gen 2 00000008:00010000:3.0:1789760780.793303:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760780.793310:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.793319:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.793338:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760780.793360:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503815296:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.795993:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760780.796014:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503815424:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.810086:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986269 gen 2 00000008:00010000:3.0:1789760780.810120:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760780.810128:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.810138:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.810156:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760780.810177:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503815424:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.810891:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503815552:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.825418:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986270 gen 2 00000008:00010000:3.0:1789760780.825462:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760780.825469:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.825478:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.825583:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760780.825609:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503815552:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.826229:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503815680:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.840115:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986271 gen 2 00000008:00010000:3.0:1789760780.840149:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760780.840156:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.840164:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.840183:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760780.840212:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503815680:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.841141:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503815808:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760780.851152:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986272 gen 2 00000008:00010000:3.0:1789760780.851189:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760780.851196:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.851206:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.851230:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760780.851265:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503815808:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760780.867142:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18966, 18966] 00000080:00200000:3.0:1789760780.867183:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.867189:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.867201:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77688832 i_blocks 151760 00000020:00200000:3.0:1789760780.867204:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.867212:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760780.867245:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77688832, count: 4096 00000080:00200000:3.0:1789760780.867252:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77688832, bytes: 4096 00000080:00200000:3.0:1789760780.867380:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760780.867388:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760780.867390:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77688832, 77692928) 0 0 00000080:00200000:3.0:1789760780.867450:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760780.867466:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18967, 18967] 00020000:00200000:3.0:1789760780.867473:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77688832 chunk: [77688832, 77692928) 77692928, 4096 00020000:00200000:3.0:1789760780.867476:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760780.867492:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18967, 18967] 00000008:00010000:3.0:1789760780.867506:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760780.867510:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760780.867519:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77688832, 77692928) append 0 append lockpos 0 00000080:00200000:3.0:1789760780.867529:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77688832=0x4a17000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760780.867572:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760780.867575:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77692928 00000020:00010000:3.0:1789760780.867581:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760780.867584:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.867587:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.867590:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760780.867593:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760780.867599:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760780.867604:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760780.867618:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760780.867869:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 77688832<>77692928 00000100:00100000:3.0:1789760780.879823:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986274 gen 2 00000008:00010000:3.0:1789760780.879859:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760780.879866:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.879874:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.879951:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760780.879976:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503816064:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760781.132205:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [18984, 18984] 00000080:00200000:3.0:1789760781.132219:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760781.132224:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760781.132249:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77762560 i_blocks 151904 00000020:00200000:3.0:1789760781.132253:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760781.132261:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760781.132284:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77762560, count: 4096 00000080:00200000:3.0:1789760781.132290:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77762560, bytes: 4096 00000080:00200000:3.0:1789760781.132299:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760781.132306:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760781.132308:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77762560, 77766656) 0 0 00000080:00200000:3.0:1789760781.132311:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760781.132322:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [18985, 18985] 00020000:00200000:3.0:1789760781.132329:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77762560 chunk: [77762560, 77766656) 77766656, 4096 00020000:00200000:3.0:1789760781.132333:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760781.132346:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [18985, 18985] 00000008:00010000:3.0:1789760781.132358:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760781.132362:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760781.132371:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77762560, 77766656) append 0 append lockpos 0 00000080:00200000:3.0:1789760781.132381:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77762560=0x4a29000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760781.132566:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760781.132569:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77766656 00000020:00010000:3.0:1789760781.132576:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760781.132579:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760781.132582:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760781.132695:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760781.132700:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760781.132810:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760781.132818:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760781.132834:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760781.132908:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 77762560<>77766656 00000100:00100000:3.0:1789760781.133012:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760781.133030:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503818368:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.141317:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986292 gen 2 00000008:00010000:3.0:1789760781.141351:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760781.141357:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.141365:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.141384:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760781.141414:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503818368:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.152183:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986293 gen 2 00000008:00010000:3.0:1789760781.152221:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760781.152227:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.152235:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.152254:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760781.152278:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503818496:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.153131:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503818624:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.163164:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986294 gen 2 00000008:00010000:3.0:1789760781.163212:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760781.163218:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.163226:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.163244:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760781.163267:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503818624:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.163822:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503818752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.174002:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986295 gen 2 00000008:00010000:3.0:1789760781.174232:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760781.174239:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.174248:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.174315:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760781.174342:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503818752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.175927:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503818880:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.185352:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986296 gen 2 00000008:00010000:3.0:1789760781.185386:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760781.185391:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.185410:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.185428:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760781.185452:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503818880:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.185993:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503819008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.195299:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986297 gen 2 00000008:00010000:3.0:1789760781.195334:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760781.195341:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.195349:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.195779:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760781.195806:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503819008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.198360:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503819136:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.207515:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986298 gen 2 00000008:00010000:3.0:1789760781.207550:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760781.207557:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.207575:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.207660:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760781.207694:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503819136:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.210828:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503819264:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.219152:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986299 gen 2 00000008:00010000:3.0:1789760781.219196:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760781.219201:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.219209:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.219224:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760781.219244:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503819264:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.220088:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503819392:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.232031:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986300 gen 2 00000008:00010000:3.0:1789760781.232096:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760781.232102:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.232110:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.232121:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760781.232143:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503819392:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.252591:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503819776:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.261561:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986303 gen 2 00000008:00010000:3.0:1789760781.261603:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760781.261609:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.261617:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.261638:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760781.261662:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503819776:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.371371:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503820928:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.383587:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986312 gen 2 00000008:00010000:3.0:1789760781.383623:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760781.383629:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.383637:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.383659:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760781.383693:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503820928:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.403734:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760781.403753:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503821184:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.506293:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503821952:192.168.203.158@tcp:4:dd.0 00010000:00010000:3.0:1789760781.631208:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760781.765460:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986334 gen 2 00000008:00010000:3.0:1789760781.765497:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760781.765503:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.765512:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.765534:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760781.765560:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503823744:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760781.791945:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19029, 19029] 00000080:00200000:3.0:1789760781.791960:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760781.791966:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760781.791978:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77946880 i_blocks 152264 00000020:00200000:3.0:1789760781.791982:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760781.791992:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760781.792017:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77946880, count: 4096 00000080:00200000:3.0:1789760781.792022:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77946880, bytes: 4096 00000080:00200000:3.0:1789760781.792030:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760781.792068:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760781.792070:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77946880, 77950976) 0 0 00000080:00200000:3.0:1789760781.792074:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760781.792086:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19030, 19030] 00020000:00200000:3.0:1789760781.792093:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77946880 chunk: [77946880, 77950976) 77950976, 4096 00020000:00200000:3.0:1789760781.792096:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760781.792111:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19030, 19030] 00000008:00010000:3.0:1789760781.792126:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760781.792131:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760781.792140:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77946880, 77950976) append 0 append lockpos 0 00000080:00200000:3.0:1789760781.792150:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77946880=0x4a56000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760781.792193:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760781.792195:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77950976 00000020:00010000:3.0:1789760781.792202:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760781.792205:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760781.792208:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760781.792211:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760781.792214:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760781.792221:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760781.792226:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760781.792239:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760781.792317:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 77946880<>77950976 00000100:00100000:3.0:1789760781.792355:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760781.792368:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503824128:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.802898:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986337 gen 2 00000008:00010000:3.0:1789760781.802936:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760781.802943:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.802951:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.802976:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760781.803002:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503824128:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760781.803850:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:3.0:1789760781.803923:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 77950976<>77955072 00000100:00100000:3.0:1789760781.814623:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986338 gen 2 00000008:00010000:3.0:1789760781.814658:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:3.0:1789760781.814673:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.814683:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.814702:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:3.0:1789760781.814727:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503824256:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760781.825541:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19032, 19032] 00000080:00200000:3.0:1789760781.825555:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760781.825560:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760781.825572:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 77959168 i_blocks 152288 00000020:00200000:3.0:1789760781.825575:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760781.825584:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760781.825611:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 77959168, count: 4096 00000080:00200000:3.0:1789760781.825617:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 77959168, bytes: 4096 00000080:00200000:3.0:1789760781.825626:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760781.825633:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760781.825634:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [77959168, 77963264) 0 0 00000080:00200000:3.0:1789760781.825638:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760781.825651:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19033, 19033] 00020000:00200000:3.0:1789760781.825658:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 77959168 chunk: [77959168, 77963264) 77963264, 4096 00020000:00200000:3.0:1789760781.825672:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760781.825688:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19033, 19033] 00000008:00010000:3.0:1789760781.825711:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760781.825716:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760781.825725:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [77959168, 77963264) append 0 append lockpos 0 00000080:00200000:3.0:1789760781.825737:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=77959168=0x4a59000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760781.825781:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760781.825784:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 77963264 00000020:00010000:3.0:1789760781.825791:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760781.825794:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760781.825797:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760781.825800:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760781.825803:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760781.825810:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760781.825815:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760781.825829:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760781.825908:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 77959168<>77963264 00000100:00100000:3.0:1789760781.825950:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760781.836893:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986340 gen 2 00000008:00010000:3.0:1789760781.836932:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760781.836941:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.836951:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.837010:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760781.837069:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503824512:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.837651:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503824640:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.846651:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986341 gen 2 00000008:00010000:3.0:1789760781.846699:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760781.846706:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.846714:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.846730:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760781.846758:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503824640:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.848095:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503824768:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.858375:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986342 gen 2 00000008:00010000:3.0:1789760781.858472:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760781.858479:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.858542:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.858606:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760781.858630:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503824768:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.859936:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503824896:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760781.873882:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986343 gen 2 00000008:00010000:3.0:1789760781.873920:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760781.873927:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.873935:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.873952:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760781.873978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503824896:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.014448:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503826432:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760782.014557:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2392:1876699503826688:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760782.014587:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:3.0:1789760782.014593:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503826304:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760782.014617:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503826560:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760782.014639:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2392:1876699503826816:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760782.111947:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986360 gen 2 00000008:00010000:3.0:1789760782.111993:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760782.112000:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.112018:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.112072:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760782.112098:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503827712:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.112695:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503827840:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.125521:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986361 gen 2 00000008:00010000:3.0:1789760782.125561:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760782.125568:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.125576:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.125594:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760782.125618:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503827840:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.135253:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986362 gen 2 00000008:00010000:3.0:1789760782.135290:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760782.135297:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.135305:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.135327:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760782.135349:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503827968:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760782.312977:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19068, 19068] 00000080:00200000:3.0:1789760782.312990:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.312996:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.313007:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78106624 i_blocks 152576 00000020:00200000:3.0:1789760782.313011:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.313018:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.313067:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78106624, count: 4096 00000080:00200000:3.0:1789760782.313074:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78106624, bytes: 4096 00000080:00200000:3.0:1789760782.313082:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.313090:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760782.313091:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78106624, 78110720) 0 0 00000080:00200000:3.0:1789760782.313095:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.313107:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19069, 19069] 00020000:00200000:3.0:1789760782.313114:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78106624 chunk: [78106624, 78110720) 78110720, 4096 00020000:00200000:3.0:1789760782.313117:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.313132:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19069, 19069] 00000008:00010000:3.0:1789760782.313145:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.313150:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760782.313159:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78106624, 78110720) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.313169:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78106624=0x4a7d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.313214:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.313217:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78110720 00000020:00010000:3.0:1789760782.313224:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.313227:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.313230:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.313234:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.313236:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.313243:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.313248:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.313263:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760782.313342:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78106624<>78110720 00000100:00100000:3.0:1789760782.313381:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760782.313642:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503829760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.320893:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986376 gen 2 00000008:00010000:3.0:1789760782.320932:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760782.320939:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.320948:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.320968:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760782.321000:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503829760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.330831:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986377 gen 2 00000008:00010000:3.0:1789760782.330867:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760782.330873:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.330882:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.330900:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760782.330924:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503829888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.331745:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503830016:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.341593:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986378 gen 2 00000008:00010000:3.0:1789760782.341630:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760782.341636:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.341645:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.341669:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760782.341692:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503830016:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.342315:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503830144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.356302:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986379 gen 2 00000008:00010000:3.0:1789760782.356338:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760782.356345:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.356353:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.356370:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760782.356392:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503830144:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760782.356886:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c480 00000008:00100000:3.0:1789760782.356965:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78123008<>78127104 00000100:00100000:3.0:1789760782.357002:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760782.357016:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503830272:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.369298:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986380 gen 2 00000008:00010000:3.0:1789760782.369333:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760782.369339:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.369348:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.369369:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760782.369395:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503830272:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.383251:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986381 gen 2 00000008:00010000:3.0:1789760782.383290:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760782.383297:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.383382:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.383401:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760782.383427:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503830400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.384850:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503830528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.397953:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986382 gen 2 00000008:00010000:3.0:1789760782.397997:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760782.398003:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.398013:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.398114:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760782.398137:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503830528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.398893:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760782.398909:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503830656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.410260:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986383 gen 2 00000008:00010000:3.0:1789760782.410300:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760782.410305:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.410314:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.410333:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760782.410356:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503830656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.411338:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503830784:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.426343:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986384 gen 2 00000008:00010000:3.0:1789760782.426381:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760782.426387:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.426396:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.426417:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760782.426444:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503830784:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.448978:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986385 gen 2 00000008:00010000:3.0:1789760782.449014:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760782.449020:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.449029:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.449087:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760782.449110:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503830912:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760782.680524:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19092, 19092] 00000080:00200000:3.0:1789760782.680538:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.680545:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.680559:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78204928 i_blocks 152768 00000020:00200000:3.0:1789760782.680563:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.680573:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.680599:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78204928, count: 4096 00000080:00200000:3.0:1789760782.680606:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78204928, bytes: 4096 00000080:00200000:3.0:1789760782.680614:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.680622:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760782.680624:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78204928, 78209024) 0 0 00000080:00200000:3.0:1789760782.680628:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.680642:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19093, 19093] 00020000:00200000:3.0:1789760782.680648:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78204928 chunk: [78204928, 78209024) 78209024, 4096 00020000:00200000:3.0:1789760782.680652:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.680679:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19093, 19093] 00000008:00010000:3.0:1789760782.680693:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.680707:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760782.680716:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78204928, 78209024) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.680726:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78204928=0x4a95000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.680770:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.680773:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78209024 00000020:00010000:3.0:1789760782.680779:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.680782:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.680785:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.680787:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.680790:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.680795:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.680801:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.680816:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760782.680892:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78204928<>78209024 00000100:00100000:3.0:1789760782.691513:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986400 gen 2 00000008:00010000:3.0:1789760782.691549:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760782.691555:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.691564:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.691586:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760782.691609:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503832832:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760782.704768:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19094, 19094] 00000080:00200000:3.0:1789760782.704779:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.704785:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.704796:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78213120 i_blocks 152784 00000020:00200000:3.0:1789760782.704799:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.704806:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.704830:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78213120, count: 4096 00000080:00200000:3.0:1789760782.704835:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78213120, bytes: 4096 00000080:00200000:3.0:1789760782.704843:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.704851:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760782.704853:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78213120, 78217216) 0 0 00000080:00200000:3.0:1789760782.704857:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.704868:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19095, 19095] 00020000:00200000:3.0:1789760782.704874:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78213120 chunk: [78213120, 78217216) 78217216, 4096 00020000:00200000:3.0:1789760782.704879:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.704892:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19095, 19095] 00000008:00010000:3.0:1789760782.704904:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.704909:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760782.704916:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78213120, 78217216) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.704927:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78213120=0x4a97000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.704966:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.704969:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78217216 00000020:00010000:3.0:1789760782.704975:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.704978:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.704981:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.704985:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.704987:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.704993:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.704998:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.705010:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760782.705498:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78213120<>78217216 00000100:00100000:3.0:1789760782.705535:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503833088:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.720301:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986402 gen 2 00000008:00010000:3.0:1789760782.720337:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760782.720343:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.720352:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.720402:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760782.720424:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503833088:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.720999:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503833216:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.728664:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986403 gen 2 00000008:00010000:3.0:1789760782.728701:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760782.728708:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.728725:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.728750:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760782.728774:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503833216:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.729433:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503833344:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.740142:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986404 gen 2 00000008:00010000:3.0:1789760782.740176:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760782.740183:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.740191:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.740209:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760782.740233:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503833344:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.742219:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760782.742237:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503833472:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.755710:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986405 gen 2 00000008:00010000:3.0:1789760782.755813:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760782.755819:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.755828:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.755844:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760782.755867:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503833472:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.757569:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503833600:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.769535:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986406 gen 2 00000008:00010000:3.0:1789760782.769572:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760782.769578:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.769588:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.769699:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760782.769957:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503833600:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.771508:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503833728:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.779772:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986407 gen 2 00000008:00010000:3.0:1789760782.779808:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760782.779815:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.779823:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.779842:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760782.779870:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503833728:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.780455:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503833856:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.788672:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986408 gen 2 00000008:00010000:3.0:1789760782.789291:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760782.789299:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.789309:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.789331:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760782.789354:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503833856:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.790020:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503833984:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.798891:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986409 gen 2 00000008:00010000:3.0:1789760782.798930:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760782.798936:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.798944:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.798962:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760782.798988:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503833984:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.810263:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986410 gen 2 00000008:00010000:3.0:1789760782.810307:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760782.810313:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.810322:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.810343:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760782.810368:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503834112:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.811144:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760782.811159:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503834240:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.819126:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986411 gen 2 00000008:00010000:3.0:1789760782.819164:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760782.819170:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.819179:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.819196:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760782.819220:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503834240:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.829349:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986412 gen 2 00000008:00010000:3.0:1789760782.829395:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760782.829402:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.829411:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.829431:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760782.829455:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503834368:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760782.969014:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19118, 19118] 00000080:00200000:3.0:1789760782.969028:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.969033:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.969075:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78311424 i_blocks 152976 00000020:00200000:3.0:1789760782.969078:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.969088:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.969113:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78311424, count: 4096 00000080:00200000:3.0:1789760782.969119:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78311424, bytes: 4096 00000080:00200000:3.0:1789760782.969126:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.969134:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760782.969136:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78311424, 78315520) 0 0 00000080:00200000:3.0:1789760782.969140:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.969152:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19119, 19119] 00020000:00200000:3.0:1789760782.969158:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78311424 chunk: [78311424, 78315520) 78315520, 4096 00020000:00200000:3.0:1789760782.969161:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.969176:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19119, 19119] 00000008:00010000:3.0:1789760782.969190:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.969194:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760782.969204:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78311424, 78315520) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.969215:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78311424=0x4aaf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.969262:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.969265:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78315520 00000020:00010000:3.0:1789760782.969272:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.969275:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.969278:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.969281:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.969283:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.969290:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.969295:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.969308:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760782.969385:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78311424<>78315520 00000100:00100000:3.0:1789760782.969424:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760782.969436:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503836160:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760782.978632:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986426 gen 2 00000008:00010000:3.0:1789760782.978680:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760782.978686:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.978695:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.978717:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760782.978742:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503836160:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760783.002936:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19121, 19121] 00000080:00200000:3.0:1789760783.002952:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.002957:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.002971:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78323712 i_blocks 153000 00000020:00200000:3.0:1789760783.002974:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.002983:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.003008:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78323712, count: 4096 00000080:00200000:3.0:1789760783.003015:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78323712, bytes: 4096 00000080:00200000:3.0:1789760783.003023:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.003031:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.003063:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78323712, 78327808) 0 0 00000080:00200000:3.0:1789760783.003067:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.003082:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19122, 19122] 00020000:00200000:3.0:1789760783.003090:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78323712 chunk: [78323712, 78327808) 78327808, 4096 00020000:00200000:3.0:1789760783.003093:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.003109:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19122, 19122] 00000008:00010000:3.0:1789760783.003124:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.003129:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.003138:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78323712, 78327808) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.003152:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78323712=0x4ab2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.003194:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.003196:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78327808 00000020:00010000:3.0:1789760783.003204:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.003207:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.003210:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.003213:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.003215:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.003222:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.003227:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.003242:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760783.003317:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78323712<>78327808 00000100:00100000:3.0:1789760783.013173:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986429 gen 2 00000008:00010000:3.0:1789760783.013216:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760783.013225:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.013234:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.013244:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760783.013267:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503836544:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760783.065323:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19127, 19127] 00000080:00200000:3.0:1789760783.065335:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.065339:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.065352:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78348288 i_blocks 153048 00000020:00200000:3.0:1789760783.065354:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.065362:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.065498:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78348288, count: 4096 00000080:00200000:3.0:1789760783.065506:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78348288, bytes: 4096 00000080:00200000:3.0:1789760783.065515:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.065523:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.065525:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78348288, 78352384) 0 0 00000080:00200000:3.0:1789760783.065529:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.065540:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19128, 19128] 00020000:00200000:3.0:1789760783.065546:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78348288 chunk: [78348288, 78352384) 78352384, 4096 00020000:00200000:3.0:1789760783.065549:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.065563:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19128, 19128] 00000008:00010000:3.0:1789760783.065577:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.065581:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.065590:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78348288, 78352384) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.065600:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78348288=0x4ab8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.065647:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.065649:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78352384 00000020:00010000:3.0:1789760783.065699:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.065704:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.065707:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.065710:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.065713:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.065720:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.065726:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.065741:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760783.065823:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78348288<>78352384 00000100:00100000:3.0:1789760783.075030:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986435 gen 2 00000008:00010000:3.0:1789760783.075102:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760783.075108:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.075116:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.075126:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760783.075148:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503837312:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760783.085144:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19129, 19129] 00000080:00200000:3.0:1789760783.085156:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.085161:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.085173:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78356480 i_blocks 153064 00000020:00200000:3.0:1789760783.085176:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.085185:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.085209:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78356480, count: 4096 00000080:00200000:3.0:1789760783.085215:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78356480, bytes: 4096 00000080:00200000:3.0:1789760783.085224:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.085231:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.085233:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78356480, 78360576) 0 0 00000080:00200000:3.0:1789760783.085236:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.085248:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19130, 19130] 00020000:00200000:3.0:1789760783.085256:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78356480 chunk: [78356480, 78360576) 78360576, 4096 00020000:00200000:3.0:1789760783.085260:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.085272:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19130, 19130] 00000008:00010000:3.0:1789760783.085295:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.085301:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.085310:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78356480, 78360576) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.085320:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78356480=0x4aba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.085361:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.085363:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78360576 00000020:00010000:3.0:1789760783.085370:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.085372:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.085375:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.085378:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.085380:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.085385:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.085390:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.085404:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760783.085480:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78356480<>78360576 00000100:00100000:3.0:1789760783.085593:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760783.085607:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503837568:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.097926:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986437 gen 2 00000008:00010000:3.0:1789760783.097963:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760783.097969:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.097977:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.097999:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760783.098023:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503837568:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760783.161192:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19135, 19135] 00000080:00200000:3.0:1789760783.161203:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.161208:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.161220:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78381056 i_blocks 153112 00000020:00200000:3.0:1789760783.161223:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.161230:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.161254:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78381056, count: 4096 00000080:00200000:3.0:1789760783.161260:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78381056, bytes: 4096 00000080:00200000:3.0:1789760783.161267:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.161275:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.161277:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78381056, 78385152) 0 0 00000080:00200000:3.0:1789760783.161281:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.161293:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19136, 19136] 00020000:00200000:3.0:1789760783.161299:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78381056 chunk: [78381056, 78385152) 78385152, 4096 00020000:00200000:3.0:1789760783.161303:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.161318:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19136, 19136] 00000008:00010000:3.0:1789760783.161333:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.161337:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.161345:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78381056, 78385152) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.161358:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78381056=0x4ac0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.161412:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.161415:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78385152 00000020:00010000:3.0:1789760783.161421:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.161424:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.161428:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.161431:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.161433:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.161439:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.161446:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.161459:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760783.161547:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78381056<>78385152 00000100:00100000:3.0:1789760783.174462:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986443 gen 2 00000008:00010000:3.0:1789760783.174805:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760783.174813:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.174821:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.174841:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760783.174863:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503838336:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760783.185410:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19137, 19137] 00000080:00200000:3.0:1789760783.185423:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.185428:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.185439:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78389248 i_blocks 153128 00000020:00200000:3.0:1789760783.185443:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.185451:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.185475:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78389248, count: 4096 00000080:00200000:3.0:1789760783.185480:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78389248, bytes: 4096 00000080:00200000:3.0:1789760783.185488:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.185613:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.185615:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78389248, 78393344) 0 0 00000080:00200000:3.0:1789760783.185618:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.185630:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19138, 19138] 00020000:00200000:3.0:1789760783.185636:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78389248 chunk: [78389248, 78393344) 78393344, 4096 00020000:00200000:3.0:1789760783.185639:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.185765:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19138, 19138] 00000008:00010000:3.0:1789760783.185782:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.185786:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.185795:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78389248, 78393344) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.185807:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78389248=0x4ac2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.185851:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.185854:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78393344 00000020:00010000:3.0:1789760783.185861:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.185865:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.185867:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.185870:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.185873:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.185879:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.185885:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.185899:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760783.185970:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78389248<>78393344 00000100:00100000:3.0:1789760783.197193:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986445 gen 2 00000008:00010000:3.0:1789760783.197231:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760783.197238:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.197247:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.197308:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760783.197333:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503838592:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760783.411729:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19153, 19153] 00000080:00200000:3.0:1789760783.411742:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.411747:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.411759:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78454784 i_blocks 153256 00000020:00200000:3.0:1789760783.411762:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.411770:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.411797:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78454784, count: 4096 00000080:00200000:3.0:1789760783.411802:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78454784, bytes: 4096 00000080:00200000:3.0:1789760783.411810:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.411817:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.411819:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78454784, 78458880) 0 0 00000080:00200000:3.0:1789760783.411823:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.411835:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19154, 19154] 00020000:00200000:3.0:1789760783.411842:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78454784 chunk: [78454784, 78458880) 78458880, 4096 00020000:00200000:3.0:1789760783.411845:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.411860:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19154, 19154] 00000008:00010000:3.0:1789760783.411873:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.411878:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.411886:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78454784, 78458880) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.411897:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78454784=0x4ad2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.411940:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.411942:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78458880 00000020:00010000:3.0:1789760783.411949:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.411953:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.411956:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.411959:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.411962:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.411968:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.411974:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.411987:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760783.413225:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78454784<>78458880 00000100:00100000:3.0:1789760783.413355:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503840640:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.422380:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986461 gen 2 00000008:00010000:3.0:1789760783.422416:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760783.422422:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.422429:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.422448:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760783.422473:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503840640:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.423065:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760783.423079:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503840768:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.443725:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986462 gen 2 00000008:00010000:3.0:1789760783.443760:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760783.443766:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.443775:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.443794:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760783.443817:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503840768:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.444632:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503840896:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.456966:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986463 gen 2 00000008:00010000:3.0:1789760783.457001:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760783.457007:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.457015:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.457107:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760783.457141:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503840896:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.460942:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760783.460965:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503841024:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.473433:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986464 gen 2 00000008:00010000:3.0:1789760783.473468:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760783.473475:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.473484:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.473505:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760783.473531:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503841024:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.474145:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503841152:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.501802:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986465 gen 2 00000008:00010000:3.0:1789760783.501835:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760783.501841:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.501850:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.501867:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760783.501886:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503841152:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.502529:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503841280:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.520543:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986466 gen 2 00000008:00010000:3.0:1789760783.520580:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760783.520586:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.520596:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.520616:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760783.520640:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503841280:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.521289:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760783.521302:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503841408:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760783.595963:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503842048:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760783.804988:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19182, 19182] 00000080:00200000:3.0:1789760783.805001:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.805006:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.805018:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78573568 i_blocks 153488 00000020:00200000:3.0:1789760783.805021:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.805029:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.805094:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78573568, count: 4096 00000080:00200000:3.0:1789760783.805100:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78573568, bytes: 4096 00000080:00200000:3.0:1789760783.805108:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.805115:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.805117:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78573568, 78577664) 0 0 00000080:00200000:3.0:1789760783.805122:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.805134:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19183, 19183] 00020000:00200000:3.0:1789760783.805141:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78573568 chunk: [78573568, 78577664) 78577664, 4096 00020000:00200000:3.0:1789760783.805145:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.805159:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19183, 19183] 00000008:00010000:3.0:1789760783.805173:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.805178:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.805187:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78573568, 78577664) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.805197:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78573568=0x4aef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.805238:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.805240:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78577664 00000020:00010000:3.0:1789760783.805247:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.805250:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.805253:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.805256:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.805258:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.805264:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.805270:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.805283:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760783.805358:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 78573568<>78577664 00000080:00200000:3.0:1789760783.836367:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19183, 19183] 00000080:00200000:3.0:1789760783.836380:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.836385:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.836397:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78577664 i_blocks 153496 00000020:00200000:3.0:1789760783.836400:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.836408:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.836434:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78577664, count: 4096 00000080:00200000:3.0:1789760783.836440:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78577664, bytes: 4096 00000080:00200000:3.0:1789760783.836448:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.836456:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.836458:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78577664, 78581760) 0 0 00000080:00200000:3.0:1789760783.836461:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.836475:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19184, 19184] 00020000:00200000:3.0:1789760783.836481:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78577664 chunk: [78577664, 78581760) 78581760, 4096 00020000:00200000:3.0:1789760783.836485:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.836500:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19184, 19184] 00000008:00010000:3.0:1789760783.836513:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.836517:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.836525:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78577664, 78581760) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.836536:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78577664=0x4af0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.836580:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.836582:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78581760 00000020:00010000:3.0:1789760783.836589:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.836593:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.836596:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.836599:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.836601:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.836607:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.836614:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.836627:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1cc0 00000008:00100000:3.0:1789760783.836721:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070d180 - object 0x280000401:2 offset 78577664<>78581760 00000080:00200000:3.0:1789760783.853939:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19184, 19184] 00000080:00200000:3.0:1789760783.853952:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.853957:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.853970:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78581760 i_blocks 153504 00000020:00200000:3.0:1789760783.853973:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.853980:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.854006:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78581760, count: 4096 00000080:00200000:3.0:1789760783.854012:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78581760, bytes: 4096 00000080:00200000:3.0:1789760783.854020:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.854027:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.854029:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78581760, 78585856) 0 0 00000080:00200000:3.0:1789760783.854058:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.854071:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19185, 19185] 00020000:00200000:3.0:1789760783.854078:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78581760 chunk: [78581760, 78585856) 78585856, 4096 00020000:00200000:3.0:1789760783.854083:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.854100:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19185, 19185] 00000008:00010000:3.0:1789760783.854115:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.854121:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.854130:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78581760, 78585856) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.854141:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78581760=0x4af1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.854178:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.854180:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78585856 00000020:00010000:3.0:1789760783.854186:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.854189:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.854192:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.854195:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.854197:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.854202:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.854208:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.870066:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19185, 19185] 00000080:00200000:3.0:1789760783.870079:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.870084:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.870095:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78585856 i_blocks 153512 00000020:00200000:3.0:1789760783.870099:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.870107:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.870131:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78585856, count: 4096 00000080:00200000:3.0:1789760783.870138:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78585856, bytes: 4096 00000080:00200000:3.0:1789760783.870147:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.870154:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.870156:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78585856, 78589952) 0 0 00000080:00200000:3.0:1789760783.870159:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.870171:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19186, 19186] 00020000:00200000:3.0:1789760783.870177:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78585856 chunk: [78585856, 78589952) 78589952, 4096 00020000:00200000:3.0:1789760783.870181:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.870195:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19186, 19186] 00000008:00010000:3.0:1789760783.870219:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.870224:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.870232:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78585856, 78589952) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.870243:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78585856=0x4af2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.870278:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.870280:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78589952 00000020:00010000:3.0:1789760783.870286:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.870289:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.870291:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.870294:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.870297:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.870303:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.870307:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.885749:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19186, 19186] 00000080:00200000:3.0:1789760783.885771:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.885776:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.885791:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78589952 i_blocks 153520 00000020:00200000:3.0:1789760783.885795:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.885808:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.885833:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78589952, count: 4096 00000080:00200000:3.0:1789760783.885841:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78589952, bytes: 4096 00000080:00200000:3.0:1789760783.885852:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.885860:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.885862:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78589952, 78594048) 0 0 00000080:00200000:3.0:1789760783.885866:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.885878:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19187, 19187] 00020000:00200000:3.0:1789760783.885884:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78589952 chunk: [78589952, 78594048) 78594048, 4096 00020000:00200000:3.0:1789760783.885888:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.885903:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19187, 19187] 00000008:00010000:3.0:1789760783.886101:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.886108:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.886118:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78589952, 78594048) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.886129:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78589952=0x4af3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.886613:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.886616:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78594048 00000020:00010000:3.0:1789760783.886623:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.886625:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.886628:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.886631:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.886633:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.886640:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.886645:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.886677:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1cc0 00000008:00100000:3.0:1789760783.886751:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ca80 - object 0x280000401:2 offset 78589952<>78594048 00000080:00200000:3.0:1789760783.898668:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19187, 19187] 00000080:00200000:3.0:1789760783.898681:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.898694:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.898712:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78594048 i_blocks 153528 00000020:00200000:3.0:1789760783.898715:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.898723:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.898759:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78594048, count: 4096 00000080:00200000:3.0:1789760783.898771:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78594048, bytes: 4096 00000080:00200000:3.0:1789760783.898780:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.898787:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.898789:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78594048, 78598144) 0 0 00000080:00200000:3.0:1789760783.898793:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.898804:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19188, 19188] 00020000:00200000:3.0:1789760783.898811:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78594048 chunk: [78594048, 78598144) 78598144, 4096 00020000:00200000:3.0:1789760783.898815:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.898829:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19188, 19188] 00000008:00010000:3.0:1789760783.898841:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.898846:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.898854:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78594048, 78598144) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.898865:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78594048=0x4af4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.898910:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.898912:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78598144 00000020:00010000:3.0:1789760783.898919:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.898922:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.898925:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.898928:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.898931:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.898937:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.898942:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.898956:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760783.899029:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070df80 - object 0x280000401:2 offset 78594048<>78598144 00000080:00200000:3.0:1789760783.909287:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19188, 19188] 00000080:00200000:3.0:1789760783.909304:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.909309:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.909321:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78598144 i_blocks 153536 00000020:00200000:3.0:1789760783.909325:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.909332:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.909356:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78598144, count: 4096 00000080:00200000:3.0:1789760783.909363:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78598144, bytes: 4096 00000080:00200000:3.0:1789760783.909370:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.909432:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760783.909434:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78598144, 78602240) 0 0 00000080:00200000:3.0:1789760783.909438:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.909451:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19189, 19189] 00020000:00200000:3.0:1789760783.909458:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78598144 chunk: [78598144, 78602240) 78602240, 4096 00020000:00200000:3.0:1789760783.909462:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.909589:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19189, 19189] 00000008:00010000:3.0:1789760783.909605:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.909610:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760783.909619:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78598144, 78602240) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.909629:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78598144=0x4af5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.909678:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.909681:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78602240 00000020:00010000:3.0:1789760783.909687:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.909849:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.909854:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.909858:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062218 (ffff9018d1a6e908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.909861:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.909869:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.909875:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.920315:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19189, 19189] 00000080:00200000:3.0:1789760783.920329:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.920334:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760783.921162:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503845248:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.290483:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503848064:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.302575:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986519 gen 2 00000008:00010000:3.0:1789760784.302613:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760784.302621:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.302629:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.302657:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760784.302683:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503848064:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.303320:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503848192:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.315091:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986520 gen 2 00000008:00010000:3.0:1789760784.315128:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760784.315134:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.315143:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.315162:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760784.315186:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503848192:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.315986:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503848320:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.327758:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986521 gen 2 00000008:00010000:3.0:1789760784.327795:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760784.327802:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.327811:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.327832:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760784.327856:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503848320:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.358011:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986522 gen 2 00000008:00010000:3.0:1789760784.358083:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760784.358091:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.358101:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.358120:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760784.358145:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503848448:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.358903:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503848576:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.369874:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986523 gen 2 00000008:00010000:3.0:1789760784.369908:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760784.369914:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.369922:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.369943:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760784.369966:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503848576:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.372246:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760784.372267:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503848704:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.386815:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986524 gen 2 00000008:00010000:3.0:1789760784.386850:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760784.386856:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.386864:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.386884:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760784.386908:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503848704:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.388922:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503848832:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.402416:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986525 gen 2 00000008:00010000:3.0:1789760784.402450:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760784.402457:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.402464:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.402522:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760784.402545:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503848832:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.403345:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503848960:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.412168:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986526 gen 2 00000008:00010000:3.0:1789760784.412215:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760784.412222:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.412230:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.412245:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760784.412268:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503848960:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.413596:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503849088:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.434804:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986527 gen 2 00000008:00010000:3.0:1789760784.434844:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760784.434850:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.434859:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.434882:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760784.434909:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503849088:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.435672:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760784.435697:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503849216:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.452387:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986528 gen 2 00000008:00010000:3.0:1789760784.452423:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760784.452430:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.452439:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.452457:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760784.452480:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503849216:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.453215:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503849344:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.470255:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986529 gen 2 00000100:00100000:3.0:1789760784.534876:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986532 gen 2 00000008:00010000:3.0:1789760784.534913:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760784.534919:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.534928:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.534950:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760784.534977:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503849728:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760784.553103:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19226, 19226] 00000080:00200000:3.0:1789760784.553115:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760784.553120:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760784.553132:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78753792 i_blocks 153840 00000020:00200000:3.0:1789760784.553136:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760784.553144:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760784.553192:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78753792, count: 4096 00000080:00200000:3.0:1789760784.553198:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78753792, bytes: 4096 00000080:00200000:3.0:1789760784.553206:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760784.553213:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760784.553215:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78753792, 78757888) 0 0 00000080:00200000:3.0:1789760784.553219:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760784.553232:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19227, 19227] 00020000:00200000:3.0:1789760784.553238:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78753792 chunk: [78753792, 78757888) 78757888, 4096 00020000:00200000:3.0:1789760784.553242:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760784.553255:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19227, 19227] 00000008:00010000:3.0:1789760784.553269:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760784.553274:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760784.553283:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78753792, 78757888) append 0 append lockpos 0 00000080:00200000:3.0:1789760784.553295:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78753792=0x4b1b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760784.553351:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760784.553354:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78757888 00000020:00010000:3.0:1789760784.553361:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062d58 (ffff9018d1a6e878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760784.553365:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062d58 (ffff9018d1a6e878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760784.553368:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760784.553371:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062d58 (ffff9018d1a6e878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760784.553374:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760784.553381:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760784.553523:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760784.553540:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1d80 00000008:00100000:3.0:1789760784.553618:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ed80 - object 0x280000401:2 offset 78753792<>78757888 00000100:00100000:3.0:1789760784.553663:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760784.553675:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503849984:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.565916:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986534 gen 2 00000008:00010000:3.0:1789760784.565952:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1d80 00000020:00200000:3.0:1789760784.565960:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.565969:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.565990:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1d80 00000100:00100000:3.0:1789760784.566015:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503849984:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.574030:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986535 gen 2 00000008:00010000:3.0:1789760784.574106:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760784.574113:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.574121:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.574132:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760784.574156:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503850112:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.594264:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503850368:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.603561:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986537 gen 2 00000008:00010000:3.0:1789760784.603598:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760784.603604:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.603613:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.603632:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760784.603665:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503850368:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.604499:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760784.604515:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503850496:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.615975:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986538 gen 2 00000080:00200000:3.0:1789760784.616229:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19231, 19231] 00000080:00200000:3.0:1789760784.616242:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760784.616247:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760784.616260:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78774272 i_blocks 153880 00000020:00200000:3.0:1789760784.616263:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760784.616271:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760784.616296:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78774272, count: 4096 00000080:00200000:3.0:1789760784.616303:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78774272, bytes: 4096 00000080:00200000:3.0:1789760784.616312:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760784.616319:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760784.616321:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78774272, 78778368) 0 0 00000080:00200000:3.0:1789760784.616325:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760784.616338:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19232, 19232] 00020000:00200000:3.0:1789760784.616346:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78774272 chunk: [78774272, 78778368) 78778368, 4096 00020000:00200000:3.0:1789760784.616350:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760784.616365:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19232, 19232] 00000008:00010000:3.0:1789760784.616378:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760784.616384:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760784.616393:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78774272, 78778368) append 0 append lockpos 0 00000080:00200000:3.0:1789760784.616405:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78774272=0x4b20000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760784.616448:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760784.616451:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78778368 00000020:00010000:3.0:1789760784.616458:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062d58 (ffff9018d1a6e878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760784.616462:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062d58 (ffff9018d1a6e878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760784.616465:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760784.616469:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062d58 (ffff9018d1a6e878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760784.616472:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760784.616478:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760784.616484:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760784.616498:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1d80 00000008:00100000:3.0:1789760784.616577:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ed80 - object 0x280000401:2 offset 78774272<>78778368 00000100:00100000:3.0:1789760784.616609:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760784.616623:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503850624:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.631908:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986539 gen 2 00000008:00010000:3.0:1789760784.631941:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1d80 00000020:00200000:3.0:1789760784.631948:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.631956:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.631977:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1d80 00000100:00100000:3.0:1789760784.632003:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503850624:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.642636:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986540 gen 2 00000008:00010000:3.0:1789760784.642746:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760784.642753:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.642763:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.642886:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760784.642912:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503850752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.660063:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503851008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.675174:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986542 gen 2 00000080:00200000:3.0:1789760784.675785:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19235, 19235] 00000080:00200000:3.0:1789760784.675798:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760784.675803:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760784.681875:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503851136:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.692361:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986543 gen 2 00000008:00010000:3.0:1789760784.692397:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760784.692403:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.692411:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.692467:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760784.692501:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503851136:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.696264:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503851264:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.706953:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986544 gen 2 00000008:00010000:3.0:1789760784.706990:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760784.706996:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.707005:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.707104:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760784.707126:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503851264:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.803366:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503852288:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760784.983536:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503853568:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760785.032452:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19257, 19257] 00000080:00200000:3.0:1789760785.032464:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.032469:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.032481:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78880768 i_blocks 154088 00000020:00200000:3.0:1789760785.032484:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.032505:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.032552:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78880768, count: 4096 00000080:00200000:3.0:1789760785.032559:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78880768, bytes: 4096 00000080:00200000:3.0:1789760785.032567:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.032573:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760785.032575:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78880768, 78884864) 0 0 00000080:00200000:3.0:1789760785.032579:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.032591:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19258, 19258] 00020000:00200000:3.0:1789760785.032597:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78880768 chunk: [78880768, 78884864) 78884864, 4096 00020000:00200000:3.0:1789760785.032600:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.032614:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19258, 19258] 00000008:00010000:3.0:1789760785.032626:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.032631:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760785.032651:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78880768, 78884864) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.032663:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78880768=0x4b3a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.032714:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.032717:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78884864 00000020:00010000:3.0:1789760785.032723:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30627b8 (ffff9018d1a6e248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.032727:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30627b8 (ffff9018d1a6e248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.032730:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.032733:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30627b8 (ffff9018d1a6e248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.032735:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.032742:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.032747:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.032761:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1d80 00000008:00100000:3.0:1789760785.032839:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ed80 - object 0x280000401:2 offset 78880768<>78884864 00000080:00200000:3.0:1789760785.048580:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19258, 19258] 00000080:00200000:3.0:1789760785.048593:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.048599:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.048611:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 78884864 i_blocks 154096 00000020:00200000:3.0:1789760785.048615:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.048623:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.048667:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 78884864, count: 4096 00000080:00200000:3.0:1789760785.048674:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 78884864, bytes: 4096 00000080:00200000:3.0:1789760785.048897:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.048905:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760785.048907:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [78884864, 78888960) 0 0 00000080:00200000:3.0:1789760785.049092:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.049106:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19259, 19259] 00020000:00200000:3.0:1789760785.049113:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 78884864 chunk: [78884864, 78888960) 78888960, 4096 00020000:00200000:3.0:1789760785.049116:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.049133:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19259, 19259] 00000008:00010000:3.0:1789760785.049146:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.049151:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760785.049160:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [78884864, 78888960) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.049171:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=78884864=0x4b3b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.049216:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.049219:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 78888960 00000020:00010000:3.0:1789760785.049225:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30627b8 (ffff9018d1a6e248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.049228:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30627b8 (ffff9018d1a6e248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.049230:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.049233:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30627b8 (ffff9018d1a6e248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.049236:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.049242:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.049248:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.049264:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f13c0 00000008:00100000:3.0:1789760785.049342:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 78884864<>78888960 00000100:00100000:3.0:1789760785.066835:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986566 gen 2 00000008:00010000:3.0:1789760785.066867:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f13c0 00000020:00200000:3.0:1789760785.066872:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.066880:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.066903:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f13c0 00000100:00100000:3.0:1789760785.066933:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503854080:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.089774:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986567 gen 2 00000080:00200000:3.0:1789760785.097232:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19260, 19260] 00000080:00200000:3.0:1789760785.097253:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.097262:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760785.098098:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760785.098116:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503854336:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.107156:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986568 gen 2 00000008:00010000:3.0:1789760785.107191:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760785.107197:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.107206:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.107232:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760785.107257:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503854336:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760785.107754:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:3.0:1789760785.107825:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 78897152<>78901248 00000100:00100000:3.0:1789760785.107859:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503854464:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.117574:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986569 gen 2 00000008:00010000:3.0:1789760785.117611:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:3.0:1789760785.117619:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.117628:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.117664:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:3.0:1789760785.117698:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503854464:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.671283:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760785.671305:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503859072:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.687297:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986605 gen 2 00000008:00010000:3.0:1789760785.687347:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760785.687356:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.687448:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.687469:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760785.687507:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503859072:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.688571:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503859200:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.707261:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986606 gen 2 00000008:00010000:3.0:1789760785.707315:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760785.707324:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.707337:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.707376:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760785.707413:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503859200:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.710280:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760785.710312:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503859328:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.719189:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986607 gen 2 00000008:00010000:3.0:1789760785.719240:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760785.719248:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.719259:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.719282:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760785.719313:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503859328:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.720300:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503859456:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.729884:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986608 gen 2 00000008:00010000:3.0:1789760785.729941:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760785.729951:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.729963:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.730080:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760785.730111:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503859456:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760785.766857:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19304, 19304] 00000080:00200000:3.0:1789760785.766874:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.766880:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.766897:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79073280 i_blocks 154464 00000020:00200000:3.0:1789760785.766901:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.766911:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.767324:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79073280, count: 4096 00000080:00200000:3.0:1789760785.767333:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79073280, bytes: 4096 00000080:00200000:3.0:1789760785.767346:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.767354:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760785.767357:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79073280, 79077376) 0 0 00000080:00200000:3.0:1789760785.767361:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.767379:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19305, 19305] 00020000:00200000:3.0:1789760785.767495:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79073280 chunk: [79073280, 79077376) 79077376, 4096 00020000:00200000:3.0:1789760785.767501:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.767525:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19305, 19305] 00000008:00010000:3.0:1789760785.767545:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.767552:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760785.767564:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79073280, 79077376) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.767579:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79073280=0x4b69000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.767648:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.767651:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79077376 00000020:00010000:3.0:1789760785.767658:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.767662:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.767665:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.767668:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.767804:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ea28/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.767816:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.767829:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.767850:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f13c0 00000008:00100000:3.0:1789760785.767951:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79073280<>79077376 00000100:00100000:3.0:1789760785.767996:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760785.768669:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503859968:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.794607:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986612 gen 2 00000008:00010000:3.0:1789760785.794711:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f13c0 00000020:00200000:3.0:1789760785.794720:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.794732:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.794789:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f13c0 00000100:00100000:3.0:1789760785.794827:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503859968:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.827627:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986613 gen 2 00000008:00010000:3.0:1789760785.827699:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760785.827708:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.827718:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.827744:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760785.827774:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503860096:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.828960:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760785.828980:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503860224:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.848387:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986614 gen 2 00000008:00010000:3.0:1789760785.848438:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760785.848447:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.848459:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.848481:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760785.848522:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503860224:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.849561:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503860352:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.868250:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986615 gen 2 00000008:00010000:3.0:1789760785.868478:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760785.868488:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.868499:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.868568:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760785.868599:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503860352:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.870994:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503860480:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.894182:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986616 gen 2 00000008:00010000:3.0:1789760785.894237:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760785.894247:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.894258:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.894460:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760785.894490:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503860480:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.899729:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503860608:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760785.913424:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986617 gen 2 00000008:00010000:3.0:1789760785.913475:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760785.913484:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.913495:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.913563:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760785.913597:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503860608:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760785.962274:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19313, 19313] 00000080:00200000:3.0:1789760785.962292:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.962299:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.962316:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79110144 i_blocks 154536 00000020:00200000:3.0:1789760785.962321:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.962331:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.962378:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79110144, count: 4096 00000080:00200000:3.0:1789760785.962385:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79110144, bytes: 4096 00000080:00200000:3.0:1789760785.962397:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.962407:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760785.962409:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79110144, 79114240) 0 0 00000080:00200000:3.0:1789760785.962414:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.962431:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19314, 19314] 00020000:00200000:3.0:1789760785.962440:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79110144 chunk: [79110144, 79114240) 79114240, 4096 00020000:00200000:3.0:1789760785.962445:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.962570:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19314, 19314] 00000008:00010000:3.0:1789760785.962591:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.962597:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760785.962608:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79110144, 79114240) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.962623:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79110144=0x4b72000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.962691:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.962694:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79114240 00000020:00010000:3.0:1789760785.962702:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.962706:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.962709:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.962713:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.962716:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ea28/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.962723:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.962730:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.962749:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f13c0 00000008:00100000:3.0:1789760785.962853:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79110144<>79114240 00000100:00100000:3.0:1789760785.980791:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986621 gen 2 00000008:00010000:3.0:1789760785.980848:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f13c0 00000020:00200000:3.0:1789760785.980858:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.980869:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.980895:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f13c0 00000100:00100000:3.0:1789760785.980928:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503861120:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760786.007119:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19316, 19316] 00000080:00200000:3.0:1789760786.007139:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.007144:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.007161:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79122432 i_blocks 154560 00000020:00200000:3.0:1789760786.007165:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.007175:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.007212:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79122432, count: 4096 00000080:00200000:3.0:1789760786.007220:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79122432, bytes: 4096 00000080:00200000:3.0:1789760786.007232:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.007241:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760786.007244:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79122432, 79126528) 0 0 00000080:00200000:3.0:1789760786.007248:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.007265:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19317, 19317] 00020000:00200000:3.0:1789760786.007274:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79122432 chunk: [79122432, 79126528) 79126528, 4096 00020000:00200000:3.0:1789760786.007278:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.007297:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19317, 19317] 00000008:00010000:3.0:1789760786.007317:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.007323:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760786.007334:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79122432, 79126528) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.007350:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79122432=0x4b75000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.007401:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.007404:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79126528 00000020:00010000:3.0:1789760786.007413:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.007417:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.007420:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.007424:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.007428:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ea28/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.007435:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.007442:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.022758:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19317, 19317] 00000080:00200000:3.0:1789760786.022774:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.022780:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.022796:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79126528 i_blocks 154568 00000020:00200000:3.0:1789760786.022799:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.022808:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.022843:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79126528, count: 4096 00000080:00200000:3.0:1789760786.022850:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79126528, bytes: 4096 00000080:00200000:3.0:1789760786.022859:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.022867:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760786.022870:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79126528, 79130624) 0 0 00000080:00200000:3.0:1789760786.022874:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.022892:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19318, 19318] 00020000:00200000:3.0:1789760786.023163:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79126528 chunk: [79126528, 79130624) 79130624, 4096 00020000:00200000:3.0:1789760786.023169:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.023191:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19318, 19318] 00000008:00010000:3.0:1789760786.023208:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.023214:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760786.023225:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79126528, 79130624) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.023240:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79126528=0x4b76000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.023310:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.023313:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79130624 00000020:00010000:3.0:1789760786.023321:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.023324:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.023327:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.023330:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.023333:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ea28/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.023342:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.023348:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760786.023366:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760786.023466:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79126528<>79130624 00000100:00100000:3.0:1789760786.023514:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760786.023532:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503861632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.039238:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986625 gen 2 00000008:00010000:3.0:1789760786.039292:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760786.039302:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.039315:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.039404:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760786.039435:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503861632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.041386:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503861760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.065307:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986626 gen 2 00000008:00010000:3.0:1789760786.065357:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760786.065365:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.065377:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.065397:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760786.065428:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503861760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.066274:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503861888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.083194:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986627 gen 2 00000008:00010000:3.0:1789760786.083248:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760786.083256:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.083268:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.083293:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760786.083335:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503861888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.084905:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.084929:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503862016:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.098606:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986628 gen 2 00000008:00010000:3.0:1789760786.098684:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ca80 00000020:00200000:3.0:1789760786.098693:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.098705:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.098916:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ca80 00000100:00100000:3.0:1789760786.098950:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503862016:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760786.240617:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19328, 19328] 00000080:00200000:3.0:1789760786.240645:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.240651:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.240670:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79171584 i_blocks 154656 00000020:00200000:3.0:1789760786.240680:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.240691:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.240727:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79171584, count: 4096 00000080:00200000:3.0:1789760786.240734:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79171584, bytes: 4096 00000080:00200000:3.0:1789760786.240745:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.240754:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760786.240756:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79171584, 79175680) 0 0 00000080:00200000:3.0:1789760786.240761:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.240779:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19329, 19329] 00020000:00200000:3.0:1789760786.240787:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79171584 chunk: [79171584, 79175680) 79175680, 4096 00020000:00200000:3.0:1789760786.240793:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.240813:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19329, 19329] 00000008:00010000:3.0:1789760786.240830:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.240836:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760786.240857:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79171584, 79175680) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.240871:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79171584=0x4b81000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.240932:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.240935:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79175680 00000020:00010000:3.0:1789760786.240944:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.240947:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.240950:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.240954:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062998 (ffff9018d1a6ea28/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.240957:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ea28/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.240965:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.240972:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760786.240989:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760786.241150:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79171584<>79175680 00000100:00100000:3.0:1789760786.241197:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760786.241214:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503863040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.259551:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986636 gen 2 00000008:00010000:3.0:1789760786.259604:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760786.259612:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.259623:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.259904:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760786.259938:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503863040:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760786.298330:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19331, 19331] 00000080:00200000:3.0:1789760786.298358:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.298366:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760786.301428:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503863424:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.316263:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986639 gen 2 00000008:00010000:3.0:1789760786.316317:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760786.316326:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.316337:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.316581:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760786.316688:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503863424:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.318071:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503863552:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.344650:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986640 gen 2 00000008:00010000:3.0:1789760786.344717:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760786.344727:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.344738:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.344766:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760786.344807:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503863552:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.346114:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.346139:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503863680:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.363901:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986641 gen 2 00000008:00010000:3.0:1789760786.363953:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760786.363962:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.363972:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.363997:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760786.364031:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503863680:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.365895:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503863808:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.386324:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986642 gen 2 00000008:00010000:3.0:1789760786.386390:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760786.386399:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.386414:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.386440:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760786.386473:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503863808:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.388069:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.388090:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503863936:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.402584:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986643 gen 2 00000008:00010000:3.0:1789760786.402639:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760786.402648:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.402658:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.402681:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760786.402712:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503863936:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760786.560912:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19343, 19343] 00000080:00200000:3.0:1789760786.560929:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.560936:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.560951:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79233024 i_blocks 154776 00000020:00200000:3.0:1789760786.560955:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.560967:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.561031:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79233024, count: 4096 00000080:00200000:3.0:1789760786.561061:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79233024, bytes: 4096 00000080:00200000:3.0:1789760786.561072:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.561081:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760786.561083:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79233024, 79237120) 0 0 00000080:00200000:3.0:1789760786.561087:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.561103:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19344, 19344] 00020000:00200000:3.0:1789760786.561113:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79233024 chunk: [79233024, 79237120) 79237120, 4096 00020000:00200000:3.0:1789760786.561117:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.561136:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19344, 19344] 00000008:00010000:3.0:1789760786.561155:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.561161:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760786.561173:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79233024, 79237120) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.561186:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79233024=0x4b90000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.561241:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.561244:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79237120 00000020:00010000:3.0:1789760786.561252:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.561255:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.561258:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.561262:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.561264:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.561271:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.561277:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760786.561365:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760786.561471:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79233024<>79237120 00000100:00100000:3.0:1789760786.562782:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760786.562819:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503864960:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.592227:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986651 gen 2 00000008:00010000:3.0:1789760786.592322:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760786.592332:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.592343:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.592371:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760786.592396:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503864960:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760786.667784:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19347, 19347] 00000080:00200000:3.0:1789760786.667798:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.667803:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.667816:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79249408 i_blocks 154808 00000020:00200000:3.0:1789760786.667820:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.667828:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.667854:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79249408, count: 4096 00000080:00200000:3.0:1789760786.667860:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79249408, bytes: 4096 00000080:00200000:3.0:1789760786.667869:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.667878:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760786.667880:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79249408, 79253504) 0 0 00000080:00200000:3.0:1789760786.667884:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.667899:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19348, 19348] 00020000:00200000:3.0:1789760786.667906:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79249408 chunk: [79249408, 79253504) 79253504, 4096 00020000:00200000:3.0:1789760786.667910:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.667925:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19348, 19348] 00000008:00010000:3.0:1789760786.667940:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.667945:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760786.667954:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79249408, 79253504) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.667965:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79249408=0x4b94000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.668012:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.668015:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79253504 00000020:00010000:3.0:1789760786.668021:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.668025:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.668028:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.668031:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.668070:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.668078:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.668084:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760786.668099:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760786.668185:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79249408<>79253504 00000100:00100000:3.0:1789760786.686938:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986655 gen 2 00000008:00010000:3.0:1789760786.686994:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760786.687003:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.687014:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.687080:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760786.687118:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503865472:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.821341:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986662 gen 2 00000008:00010000:3.0:1789760786.821399:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760786.821407:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.821419:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.821485:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760786.821516:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503866368:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.822359:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503866496:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.836243:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986663 gen 2 00000008:00010000:3.0:1789760786.836296:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760786.836304:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.836316:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.836337:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760786.836368:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503866496:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.840470:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503866624:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.856104:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986664 gen 2 00000008:00010000:3.0:1789760786.856161:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760786.856170:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.856182:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.856208:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760786.856238:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503866624:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.859026:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.859085:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503866752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.876100:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986665 gen 2 00000008:00010000:3.0:1789760786.876160:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760786.876169:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.876179:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.876202:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760786.876231:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503866752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.877772:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503866880:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.897164:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986666 gen 2 00000008:00010000:3.0:1789760786.897226:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760786.897234:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.897245:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.897264:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760786.897297:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503866880:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.898555:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.898579:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.917876:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986667 gen 2 00000008:00010000:3.0:1789760786.917924:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760786.917932:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.917943:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.917965:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760786.917996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.918903:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867136:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.933736:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986668 gen 2 00000008:00010000:3.0:1789760786.933790:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760786.933800:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.933822:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.933841:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760786.933872:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867136:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.934604:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503867264:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.944967:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986669 gen 2 00000008:00010000:3.0:1789760786.945004:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760786.945010:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.945017:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.945072:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760786.945096:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503867264:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.946615:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867392:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.964426:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986670 gen 2 00000008:00010000:3.0:1789760786.964482:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760786.964490:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.964502:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.964531:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760786.964563:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867392:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.966030:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.966072:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867520:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.979163:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986671 gen 2 00000008:00010000:3.0:1789760786.979211:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760786.979220:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.979230:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.979288:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760786.979317:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867520:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.980512:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867648:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.989415:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986672 gen 2 00000008:00010000:3.0:1789760786.989453:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760786.989460:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.989468:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.989637:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760786.989663:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867648:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760786.991455:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.991473:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867776:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.001255:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986673 gen 2 00000008:00010000:3.0:1789760787.001293:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760787.001299:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.001308:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.001327:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760787.001355:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503867776:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.001865:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760787.001879:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503867904:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.010508:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986674 gen 2 00000008:00010000:3.0:1789760787.010545:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760787.010561:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.010570:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.010588:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760787.010612:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503867904:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.011375:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760787.011390:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503868032:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.021120:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986675 gen 2 00000008:00010000:3.0:1789760787.021156:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760787.021163:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.021171:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.021190:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760787.021211:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503868032:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.022282:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503868160:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.036872:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986676 gen 2 00000008:00010000:3.0:1789760787.036908:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760787.036915:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.036923:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.036947:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760787.036970:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503868160:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760787.068030:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19371, 19371] 00000080:00200000:3.0:1789760787.068530:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.068535:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.068549:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79347712 i_blocks 155000 00000020:00200000:3.0:1789760787.068553:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.068561:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.068588:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79347712, count: 4096 00000080:00200000:3.0:1789760787.068594:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79347712, bytes: 4096 00000080:00200000:3.0:1789760787.068603:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.068610:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760787.068612:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79347712, 79351808) 0 0 00000080:00200000:3.0:1789760787.068616:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.069849:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19372, 19372] 00020000:00200000:3.0:1789760787.069860:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79347712 chunk: [79347712, 79351808) 79351808, 4096 00020000:00200000:3.0:1789760787.069865:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.069884:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19372, 19372] 00000008:00010000:3.0:1789760787.069898:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.069903:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760787.069912:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79347712, 79351808) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.069923:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79347712=0x4bac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.069975:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.069978:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79351808 00000020:00010000:3.0:1789760787.069985:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.069988:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.069990:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.069993:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.069996:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6e758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.070003:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.070009:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.070030:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760787.070319:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79347712<>79351808 00000100:00100000:3.0:1789760787.084829:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986679 gen 2 00000008:00010000:3.0:1789760787.084865:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:3.0:1789760787.084872:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.084880:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.084896:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:3.0:1789760787.084920:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503868544:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.086029:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503868672:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.097498:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986680 gen 2 00000008:00010000:3.0:1789760787.097537:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760787.097543:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.097552:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.097572:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760787.097595:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503868672:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.120491:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986681 gen 2 00000008:00010000:3.0:1789760787.120525:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760787.120531:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.120539:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.120561:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760787.120583:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503868800:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.134510:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503869056:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760787.136296:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760787.136311:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503869184:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760787.141861:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503869568:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760787.141930:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1789760787.141938:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503869440:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760787.141966:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503869696:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760787.148469:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503869184:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760787.148487:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503869056:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760787.154464:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503869440:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760787.155219:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503869696:192.168.203.158@tcp:400:kworker.0 00000020:00200000:1.0:1789760787.334822:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.334829:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.334844:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:1.0:1789760787.334867:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503871488:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760787.345375:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19391, 19391] 00000080:00200000:1.0:1789760787.345387:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.345392:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.345404:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79429632 i_blocks 155160 00000020:00200000:1.0:1789760787.345407:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.345414:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.345439:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79429632, count: 4096 00000080:00200000:1.0:1789760787.345445:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79429632, bytes: 4096 00000080:00200000:1.0:1789760787.345454:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.345461:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.345462:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79429632, 79433728) 0 0 00000080:00200000:1.0:1789760787.345466:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.345478:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19392, 19392] 00020000:00200000:1.0:1789760787.345485:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79429632 chunk: [79429632, 79433728) 79433728, 4096 00020000:00200000:1.0:1789760787.345488:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.345503:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19392, 19392] 00000008:00010000:1.0:1789760787.345517:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.345522:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.345530:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79429632, 79433728) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.345540:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79429632=0x4bc0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.345583:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.345586:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79433728 00000020:00010000:1.0:1789760787.345592:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.345596:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.345598:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.345601:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.345603:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6e758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.345610:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.345615:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.345638:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760787.345793:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 79429632<>79433728 00000080:00200000:1.0:1789760787.359294:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19392, 19392] 00000080:00200000:1.0:1789760787.359306:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.359311:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.359323:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79433728 i_blocks 155168 00000020:00200000:1.0:1789760787.359327:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.359335:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.359372:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79433728, count: 4096 00000080:00200000:1.0:1789760787.359379:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79433728, bytes: 4096 00000080:00200000:1.0:1789760787.359388:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.359395:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.359397:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79433728, 79437824) 0 0 00000080:00200000:1.0:1789760787.359401:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.359414:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19393, 19393] 00020000:00200000:1.0:1789760787.359421:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79433728 chunk: [79433728, 79437824) 79437824, 4096 00020000:00200000:1.0:1789760787.359424:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.359439:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19393, 19393] 00000008:00010000:1.0:1789760787.359463:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.359469:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.359478:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79433728, 79437824) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.359488:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79433728=0x4bc1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.359552:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.359555:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79437824 00000020:00010000:1.0:1789760787.359562:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.359565:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.359567:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.359571:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.359573:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6e758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.359580:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.359585:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.359599:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760787.359912:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 79433728<>79437824 00000008:00010000:1.0:1789760787.370771:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:1.0:1789760787.370779:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.370788:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.370857:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:1.0:1789760787.370884:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503871872:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760787.370962:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19393, 19393] 00000080:00200000:3.0:1789760787.370975:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.370980:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.371093:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79437824 i_blocks 155176 00000020:00200000:1.0:1789760787.371098:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.371106:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.371132:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79437824, count: 4096 00000080:00200000:1.0:1789760787.371137:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79437824, bytes: 4096 00000080:00200000:1.0:1789760787.371147:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.371154:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.371156:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79437824, 79441920) 0 0 00000080:00200000:1.0:1789760787.371160:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.371175:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19394, 19394] 00020000:00200000:1.0:1789760787.371182:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79437824 chunk: [79437824, 79441920) 79441920, 4096 00020000:00200000:1.0:1789760787.371186:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.371200:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19394, 19394] 00000008:00010000:1.0:1789760787.371215:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.371219:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.371228:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79437824, 79441920) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.371238:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79437824=0x4bc2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.371349:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.371352:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79441920 00000020:00010000:1.0:1789760787.371359:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.371362:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.371365:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.371369:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.371371:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6e758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.371377:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.371382:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.371588:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760787.371791:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 79437824<>79441920 00000100:00100000:1.0:1789760787.371831:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760787.371845:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503872000:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760787.381932:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986701 gen 2 00000008:00010000:1.0:1789760787.381970:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:1.0:1789760787.381976:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.381985:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.382128:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:1.0:1789760787.382152:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503872000:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760787.383066:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503872128:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760787.394329:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986702 gen 2 00000008:00010000:1.0:1789760787.394364:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760787.394371:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.394380:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.394555:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760787.394583:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503872128:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.395394:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503872256:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.403649:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986703 gen 2 00000008:00010000:3.0:1789760787.403801:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760787.403808:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.403817:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.403837:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760787.403863:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503872256:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.405064:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503872384:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.422178:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986704 gen 2 00000008:00010000:3.0:1789760787.422214:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760787.422220:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.422229:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.422307:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760787.422330:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503872384:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.424950:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760787.424966:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503872512:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.439338:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986705 gen 2 00000008:00010000:3.0:1789760787.439375:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760787.439382:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.439391:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.440329:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760787.440347:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503872512:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.453819:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986706 gen 2 00000008:00010000:3.0:1789760787.453855:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760787.453861:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.453869:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.453892:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760787.454541:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503872640:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760787.454757:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c900 00000008:00100000:3.0:1789760787.454831:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79462400<>79466496 00000100:00100000:3.0:1789760787.454863:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503872768:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.463761:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986707 gen 2 00000008:00010000:3.0:1789760787.463803:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760787.463810:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.463820:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.463879:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760787.463904:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503872768:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.464449:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503872896:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.485686:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986708 gen 2 00000008:00010000:3.0:1789760787.485722:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760787.485728:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.485737:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.485754:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760787.485777:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503872896:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.486485:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503873024:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.496477:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986709 gen 2 00000008:00010000:3.0:1789760787.496516:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760787.496524:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.496533:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.496756:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760787.496782:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503873024:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.499851:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760787.499871:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503873152:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.521648:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986710 gen 2 00000008:00010000:3.0:1789760787.521695:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760787.521702:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.521710:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.521733:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760787.521758:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503873152:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760787.522070:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19403, 19403] 00000080:00200000:1.0:1789760787.522084:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.522089:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.522101:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79478784 i_blocks 155256 00000020:00200000:1.0:1789760787.522104:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.522113:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.522149:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79478784, count: 4096 00000080:00200000:1.0:1789760787.522156:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79478784, bytes: 4096 00000080:00200000:1.0:1789760787.522164:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.522172:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.522174:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79478784, 79482880) 0 0 00000080:00200000:1.0:1789760787.522177:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.522190:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19404, 19404] 00020000:00200000:1.0:1789760787.522197:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79478784 chunk: [79478784, 79482880) 79482880, 4096 00020000:00200000:1.0:1789760787.522201:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.522218:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19404, 19404] 00000008:00010000:1.0:1789760787.522232:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.522238:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.522246:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79478784, 79482880) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.522258:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79478784=0x4bcc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.522306:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.522309:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79482880 00000020:00010000:1.0:1789760787.522316:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.522319:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.522322:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.522325:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c30625d8 (ffff9018d1a6e758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.522327:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6e758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.522334:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.522340:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.522356:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760787.522429:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 79478784<>79482880 00000100:00100000:1.0:1789760787.522466:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760787.522480:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503873280:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760787.546438:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986711 gen 2 00000008:00010000:1.0:1789760787.546475:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:1.0:1789760787.546482:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.546492:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.546516:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:1.0:1789760787.546541:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503873280:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760787.546651:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19404, 19404] 00000080:00200000:3.0:1789760787.546664:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.546670:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.546681:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79482880 i_blocks 155264 00000020:00200000:3.0:1789760787.546685:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.546693:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.546741:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79482880, count: 4096 00000080:00200000:3.0:1789760787.546748:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79482880, bytes: 4096 00000080:00200000:3.0:1789760787.546756:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.546763:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760787.546765:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79482880, 79486976) 0 0 00000080:00200000:3.0:1789760787.546769:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.546782:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19405, 19405] 00020000:00200000:3.0:1789760787.546788:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79482880 chunk: [79482880, 79486976) 79486976, 4096 00020000:00200000:3.0:1789760787.546792:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.546807:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19405, 19405] 00000008:00010000:3.0:1789760787.546831:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.546837:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760787.546846:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79482880, 79486976) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.546856:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79482880=0x4bcd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.546892:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.546894:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79486976 00000020:00010000:3.0:1789760787.546899:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.546902:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.546905:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.546908:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.546911:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.546930:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.546936:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.547396:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:1.0:1789760787.547475:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 79482880<>79486976 00000100:00100000:1.0:1789760787.547508:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503873408:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760787.557763:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986712 gen 2 00000008:00010000:1.0:1789760787.557800:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:1.0:1789760787.557806:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.557815:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.557835:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:1.0:1789760787.557861:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503873408:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760787.557954:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19405, 19405] 00000080:00200000:3.0:1789760787.557966:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.557971:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.557984:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79486976 i_blocks 155272 00000020:00200000:3.0:1789760787.557987:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.557995:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.558018:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79486976, count: 4096 00000080:00200000:3.0:1789760787.558023:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79486976, bytes: 4096 00000080:00200000:3.0:1789760787.558031:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.558074:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760787.558076:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79486976, 79491072) 0 0 00000080:00200000:3.0:1789760787.558079:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.558091:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19406, 19406] 00020000:00200000:3.0:1789760787.558097:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79486976 chunk: [79486976, 79491072) 79491072, 4096 00020000:00200000:3.0:1789760787.558101:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.558115:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19406, 19406] 00000008:00010000:3.0:1789760787.558129:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.558134:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760787.558142:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79486976, 79491072) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.558153:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79486976=0x4bce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.558195:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.558198:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79491072 00000020:00010000:3.0:1789760787.558204:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.558208:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.558211:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.558214:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.558217:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.558222:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.558228:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.558242:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760787.558317:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79486976<>79491072 00000100:00100000:3.0:1789760787.558368:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760787.558381:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503873536:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.571630:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986713 gen 2 00000008:00010000:3.0:1789760787.571668:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760787.571674:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.571682:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.571701:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760787.571723:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503873536:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760787.571904:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19406, 19406] 00000080:00200000:1.0:1789760787.571916:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.571921:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.571933:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79491072 i_blocks 155280 00000020:00200000:1.0:1789760787.571935:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.571944:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.571967:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79491072, count: 4096 00000080:00200000:1.0:1789760787.571973:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79491072, bytes: 4096 00000080:00200000:1.0:1789760787.571981:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.571988:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.571989:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79491072, 79495168) 0 0 00000080:00200000:1.0:1789760787.571993:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.572004:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19407, 19407] 00020000:00200000:1.0:1789760787.572010:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79491072 chunk: [79491072, 79495168) 79495168, 4096 00020000:00200000:1.0:1789760787.572013:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.572027:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19407, 19407] 00000008:00010000:1.0:1789760787.572067:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.572073:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.572080:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79491072, 79495168) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.572090:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79491072=0x4bcf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.572130:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.572132:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79495168 00000020:00010000:1.0:1789760787.572138:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.572141:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.572143:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.572146:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.572148:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.572154:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.572159:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.572170:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760787.572251:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 79491072<>79495168 00000080:00200000:1.0:1789760787.584019:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19407, 19407] 00000080:00200000:1.0:1789760787.584031:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.584055:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.584066:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79495168 i_blocks 155288 00000020:00200000:1.0:1789760787.584069:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.584077:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.584099:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79495168, count: 4096 00000080:00200000:1.0:1789760787.584104:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79495168, bytes: 4096 00000080:00200000:1.0:1789760787.584110:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.584117:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.584119:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79495168, 79499264) 0 0 00000080:00200000:1.0:1789760787.584123:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.584135:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19408, 19408] 00020000:00200000:1.0:1789760787.584141:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79495168 chunk: [79495168, 79499264) 79499264, 4096 00020000:00200000:1.0:1789760787.584145:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.584159:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19408, 19408] 00000008:00010000:1.0:1789760787.584170:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.584174:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.584183:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79495168, 79499264) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.584192:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79495168=0x4bd0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.584230:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.584233:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79499264 00000020:00010000:1.0:1789760787.584239:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.584242:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.584245:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.584248:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.584251:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.584256:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.584260:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.584272:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760787.585472:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 79495168<>79499264 00000080:00200000:1.0:1789760787.603570:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19408, 19408] 00000080:00200000:1.0:1789760787.603656:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.603661:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.603672:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79499264 i_blocks 155296 00000020:00200000:1.0:1789760787.603675:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.603682:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.603705:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79499264, count: 4096 00000080:00200000:1.0:1789760787.603710:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79499264, bytes: 4096 00000080:00200000:1.0:1789760787.603718:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.603726:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.603728:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79499264, 79503360) 0 0 00000080:00200000:1.0:1789760787.603731:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.603743:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19409, 19409] 00020000:00200000:1.0:1789760787.603750:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79499264 chunk: [79499264, 79503360) 79503360, 4096 00020000:00200000:1.0:1789760787.603754:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.603768:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19409, 19409] 00000008:00010000:1.0:1789760787.603780:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.603785:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.603794:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79499264, 79503360) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.603803:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79499264=0x4bd1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.603848:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.603851:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79503360 00000020:00010000:1.0:1789760787.603857:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.603860:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.603863:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.603866:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.603869:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.603875:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.603880:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.603894:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760787.603969:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 79499264<>79503360 00000080:00200000:1.0:1789760787.620580:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19409, 19409] 00000080:00200000:1.0:1789760787.620593:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.620598:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.620610:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79503360 i_blocks 155304 00000020:00200000:1.0:1789760787.620614:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.620630:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.620656:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79503360, count: 4096 00000080:00200000:1.0:1789760787.620670:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79503360, bytes: 4096 00000080:00200000:1.0:1789760787.620677:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.620683:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.620685:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79503360, 79507456) 0 0 00000080:00200000:1.0:1789760787.620688:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.620699:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19410, 19410] 00020000:00200000:1.0:1789760787.620705:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79503360 chunk: [79503360, 79507456) 79507456, 4096 00020000:00200000:1.0:1789760787.620708:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.620722:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19410, 19410] 00000008:00010000:1.0:1789760787.620734:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.620739:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.620747:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79503360, 79507456) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.620756:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79503360=0x4bd2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.620797:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.620800:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79507456 00000020:00010000:1.0:1789760787.620806:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.620809:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.620812:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.620816:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.620818:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.620823:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.620829:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.620842:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58cc0 00000008:00100000:1.0:1789760787.620912:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 79503360<>79507456 00000080:00200000:1.0:1789760787.642985:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19410, 19410] 00000080:00200000:1.0:1789760787.642997:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.643002:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.643014:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79507456 i_blocks 155312 00000020:00200000:1.0:1789760787.643018:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.643025:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.643069:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79507456, count: 4096 00000080:00200000:1.0:1789760787.643076:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79507456, bytes: 4096 00000080:00200000:1.0:1789760787.643083:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.643091:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.643093:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79507456, 79511552) 0 0 00000080:00200000:1.0:1789760787.643096:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.643108:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19411, 19411] 00020000:00200000:1.0:1789760787.643114:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79507456 chunk: [79507456, 79511552) 79511552, 4096 00020000:00200000:1.0:1789760787.643118:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.643131:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19411, 19411] 00000008:00010000:1.0:1789760787.643144:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.643150:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.643160:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79507456, 79511552) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.643170:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79507456=0x4bd3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.643212:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.643215:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79511552 00000020:00010000:1.0:1789760787.643221:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.643224:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.643227:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.643230:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.643233:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.643240:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.643246:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.643259:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760787.643334:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 79507456<>79511552 00000080:00200000:1.0:1789760787.653961:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19411, 19411] 00000080:00200000:1.0:1789760787.653973:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.653979:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.653990:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79511552 i_blocks 155320 00000020:00200000:1.0:1789760787.653993:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.654000:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.654026:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79511552, count: 4096 00000080:00200000:1.0:1789760787.654032:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79511552, bytes: 4096 00000080:00200000:1.0:1789760787.654088:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.654095:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.654097:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79511552, 79515648) 0 0 00000080:00200000:1.0:1789760787.654100:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.654113:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19412, 19412] 00020000:00200000:1.0:1789760787.654119:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79511552 chunk: [79511552, 79515648) 79515648, 4096 00020000:00200000:1.0:1789760787.654122:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.654136:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19412, 19412] 00000008:00010000:1.0:1789760787.654149:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.654154:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.654163:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79511552, 79515648) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.654173:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79511552=0x4bd4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.654212:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.654215:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79515648 00000020:00010000:1.0:1789760787.654221:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.654224:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.654227:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.654229:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.654232:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.654238:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.654243:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.654253:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760787.654326:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 79511552<>79515648 00000080:00200000:1.0:1789760787.669588:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19412, 19412] 00000080:00200000:1.0:1789760787.669601:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.669605:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.669617:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79515648 i_blocks 155328 00000020:00200000:1.0:1789760787.669631:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.669639:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.669677:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79515648, count: 4096 00000080:00200000:1.0:1789760787.669683:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79515648, bytes: 4096 00000080:00200000:1.0:1789760787.669691:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.669699:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.669700:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79515648, 79519744) 0 0 00000080:00200000:1.0:1789760787.669704:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.669716:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19413, 19413] 00020000:00200000:1.0:1789760787.669723:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79515648 chunk: [79515648, 79519744) 79519744, 4096 00020000:00200000:1.0:1789760787.669726:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.670470:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19413, 19413] 00000008:00010000:1.0:1789760787.670488:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.670504:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.670513:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79515648, 79519744) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.670527:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79515648=0x4bd5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.670580:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.670584:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79519744 00000020:00010000:1.0:1789760787.670590:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.670593:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.670596:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.670599:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.670601:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.670608:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.670613:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.670640:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760787.670730:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 79515648<>79519744 00000080:00200000:1.0:1789760787.683427:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19413, 19413] 00000080:00200000:1.0:1789760787.683440:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.683444:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.683457:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79519744 i_blocks 155336 00000020:00200000:1.0:1789760787.683460:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.683468:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.683491:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79519744, count: 4096 00000080:00200000:1.0:1789760787.683497:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79519744, bytes: 4096 00000080:00200000:1.0:1789760787.683505:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.683512:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.683514:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79519744, 79523840) 0 0 00000080:00200000:1.0:1789760787.683518:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.683530:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19414, 19414] 00020000:00200000:1.0:1789760787.683536:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79519744 chunk: [79519744, 79523840) 79523840, 4096 00020000:00200000:1.0:1789760787.683540:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.683555:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19414, 19414] 00000008:00010000:1.0:1789760787.683569:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.683574:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.683583:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79519744, 79523840) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.683595:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79519744=0x4bd6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.683652:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.683655:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79523840 00000020:00010000:1.0:1789760787.683670:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.683673:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.683676:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.683679:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.683682:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.683688:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.683694:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.683708:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760787.683784:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 79519744<>79523840 00000080:00200000:1.0:1789760787.695171:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19414, 19414] 00000080:00200000:1.0:1789760787.695184:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.695189:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.695201:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79523840 i_blocks 155344 00000020:00200000:1.0:1789760787.695205:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.695213:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.695249:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79523840, count: 4096 00000080:00200000:1.0:1789760787.695256:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79523840, bytes: 4096 00000080:00200000:1.0:1789760787.695264:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.695271:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.695273:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79523840, 79527936) 0 0 00000080:00200000:1.0:1789760787.695278:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.695291:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19415, 19415] 00020000:00200000:1.0:1789760787.695297:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79523840 chunk: [79523840, 79527936) 79527936, 4096 00020000:00200000:1.0:1789760787.695301:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.695314:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19415, 19415] 00000008:00010000:1.0:1789760787.695327:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.695332:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.695340:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79523840, 79527936) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.695351:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79523840=0x4bd7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.695591:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.695594:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79527936 00000020:00010000:1.0:1789760787.695601:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.695604:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.695608:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.695611:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.695614:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.695633:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.695639:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.695652:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760787.695798:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 79523840<>79527936 00000080:00200000:1.0:1789760787.719834:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19415, 19415] 00000080:00200000:1.0:1789760787.719846:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.719851:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.719864:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79527936 i_blocks 155352 00000020:00200000:1.0:1789760787.719867:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.719875:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.719908:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79527936, count: 4096 00000080:00200000:1.0:1789760787.719914:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79527936, bytes: 4096 00000080:00200000:1.0:1789760787.719921:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.719928:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.719930:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79527936, 79532032) 0 0 00000080:00200000:1.0:1789760787.719934:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.719946:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19416, 19416] 00020000:00200000:1.0:1789760787.719953:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79527936 chunk: [79527936, 79532032) 79532032, 4096 00020000:00200000:1.0:1789760787.719956:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.719970:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19416, 19416] 00000008:00010000:1.0:1789760787.719983:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.719988:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.719996:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79527936, 79532032) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.720007:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79527936=0x4bd8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.720080:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.720083:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79532032 00000020:00010000:1.0:1789760787.720089:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.720092:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.720094:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.720097:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.720100:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.720106:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.720111:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.720123:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760787.720197:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 79527936<>79532032 00000100:00100000:1.0:1789760787.720231:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760787.720243:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503874816:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760787.741461:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986723 gen 2 00000008:00010000:1.0:1789760787.741498:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:1.0:1789760787.741505:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.741514:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.741536:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:1.0:1789760787.741558:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503874816:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760787.741650:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19416, 19416] 00000080:00200000:3.0:1789760787.741662:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.741667:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.741678:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79532032 i_blocks 155360 00000020:00200000:3.0:1789760787.741681:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.741690:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.741714:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79532032, count: 4096 00000080:00200000:3.0:1789760787.741720:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79532032, bytes: 4096 00000080:00200000:3.0:1789760787.741728:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.741734:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760787.741736:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79532032, 79536128) 0 0 00000080:00200000:3.0:1789760787.741740:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.741753:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19417, 19417] 00020000:00200000:3.0:1789760787.741760:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79532032 chunk: [79532032, 79536128) 79536128, 4096 00020000:00200000:3.0:1789760787.741764:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.741778:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19417, 19417] 00000008:00010000:3.0:1789760787.741792:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.741797:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760787.741807:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79532032, 79536128) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.741817:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79532032=0x4bd9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.741859:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.741862:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79536128 00000020:00010000:3.0:1789760787.741868:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.741871:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.741874:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.741877:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.741880:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.741886:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.741892:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.741906:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760787.741984:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79532032<>79536128 00000100:00100000:3.0:1789760787.751987:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986724 gen 2 00000008:00010000:3.0:1789760787.752024:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760787.752031:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.752080:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.752092:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760787.752115:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503874944:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760787.763511:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19418, 19418] 00000080:00200000:3.0:1789760787.763523:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.763528:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.763540:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79540224 i_blocks 155376 00000020:00200000:3.0:1789760787.763544:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.763552:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.763577:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79540224, count: 4096 00000080:00200000:3.0:1789760787.763582:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79540224, bytes: 4096 00000080:00200000:3.0:1789760787.763589:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.763597:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760787.763599:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79540224, 79544320) 0 0 00000080:00200000:3.0:1789760787.763603:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.763614:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19419, 19419] 00020000:00200000:3.0:1789760787.763631:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79540224 chunk: [79540224, 79544320) 79544320, 4096 00020000:00200000:3.0:1789760787.763635:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.763650:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19419, 19419] 00000008:00010000:3.0:1789760787.763767:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.763772:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760787.763781:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79540224, 79544320) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.763792:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79540224=0x4bdb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.763832:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.763834:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79544320 00000020:00010000:3.0:1789760787.763840:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.763852:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.763856:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.763859:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.763861:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.763868:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.763873:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.763886:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760787.763957:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79540224<>79544320 00000100:00100000:3.0:1789760787.777454:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986726 gen 2 00000008:00010000:3.0:1789760787.777732:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760787.777740:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.777749:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.777810:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760787.777836:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503875200:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760787.780454:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760787.780528:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503875328:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760787.791852:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986727 gen 2 00000008:00010000:1.0:1789760787.791887:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760787.791894:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.791902:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.791960:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760787.791986:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503875328:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760787.792464:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19420, 19420] 00000080:00200000:3.0:1789760787.792477:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.792482:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.792566:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79548416 i_blocks 155392 00000020:00200000:1.0:1789760787.792570:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.792579:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.792799:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79548416, count: 4096 00000080:00200000:1.0:1789760787.792807:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79548416, bytes: 4096 00000080:00200000:1.0:1789760787.792820:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.792827:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.792829:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79548416, 79552512) 0 0 00000080:00200000:1.0:1789760787.792833:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.792849:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19421, 19421] 00020000:00200000:1.0:1789760787.792856:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79548416 chunk: [79548416, 79552512) 79552512, 4096 00020000:00200000:1.0:1789760787.792860:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.792877:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19421, 19421] 00000008:00010000:1.0:1789760787.792892:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.792897:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.792907:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79548416, 79552512) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.792918:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79548416=0x4bdd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.792974:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.792977:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79552512 00000020:00010000:1.0:1789760787.792984:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.792987:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.792989:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.792992:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.792994:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.793000:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.793005:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.793062:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760787.793206:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 79548416<>79552512 00000100:00100000:3.0:1789760787.804387:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503875584:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.823353:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986729 gen 2 00000008:00010000:3.0:1789760787.823389:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760787.823396:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.823405:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.823476:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760787.823499:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503875584:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.824633:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503875712:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.833017:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986730 gen 2 00000008:00010000:3.0:1789760787.833080:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760787.833087:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.833094:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.833111:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760787.833665:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503875712:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760787.833700:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cd80 00000008:00100000:3.0:1789760787.833770:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79560704<>79564800 00000100:00100000:3.0:1789760787.833803:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760787.833815:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503875840:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.846114:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986731 gen 2 00000008:00010000:3.0:1789760787.846220:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760787.846227:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.846235:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.846254:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760787.846277:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503875840:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760787.865801:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986732 gen 2 00000008:00010000:3.0:1789760787.865839:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760787.865845:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.865854:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.865877:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760787.865902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503875968:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760787.866059:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19425, 19425] 00000080:00200000:1.0:1789760787.866071:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.866076:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.866088:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79568896 i_blocks 155432 00000020:00200000:1.0:1789760787.866091:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.866099:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.866123:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79568896, count: 4096 00000080:00200000:1.0:1789760787.866128:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79568896, bytes: 4096 00000080:00200000:1.0:1789760787.866147:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.866154:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.866156:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79568896, 79572992) 0 0 00000080:00200000:1.0:1789760787.866159:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.866170:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19426, 19426] 00020000:00200000:1.0:1789760787.866177:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79568896 chunk: [79568896, 79572992) 79572992, 4096 00020000:00200000:1.0:1789760787.866181:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.866194:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19426, 19426] 00000008:00010000:1.0:1789760787.866208:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.866212:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.866221:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79568896, 79572992) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.866230:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79568896=0x4be2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.866269:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.866271:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79572992 00000020:00010000:1.0:1789760787.866277:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.866280:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.866283:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.866286:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.866289:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.866294:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.866300:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.866311:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760787.866391:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 79568896<>79572992 00000080:00200000:1.0:1789760787.887969:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19426, 19426] 00000080:00200000:1.0:1789760787.887982:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.887987:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.887999:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79572992 i_blocks 155440 00000020:00200000:1.0:1789760787.888015:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.888023:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.888079:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79572992, count: 4096 00000080:00200000:1.0:1789760787.888089:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79572992, bytes: 4096 00000080:00200000:1.0:1789760787.888096:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.888104:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.888106:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79572992, 79577088) 0 0 00000080:00200000:1.0:1789760787.888109:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.888123:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19427, 19427] 00020000:00200000:1.0:1789760787.888130:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79572992 chunk: [79572992, 79577088) 79577088, 4096 00020000:00200000:1.0:1789760787.888138:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.888157:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19427, 19427] 00000008:00010000:1.0:1789760787.888173:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.888178:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.888186:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79572992, 79577088) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.888197:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79572992=0x4be3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.888244:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.888246:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79577088 00000020:00010000:1.0:1789760787.888252:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.888256:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.888258:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.888261:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.888264:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.888269:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.888275:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.888523:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760787.888613:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 79572992<>79577088 00000080:00200000:1.0:1789760787.906322:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19427, 19427] 00000080:00200000:1.0:1789760787.906338:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.906344:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.906356:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79577088 i_blocks 155448 00000020:00200000:1.0:1789760787.906361:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.906370:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.906394:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79577088, count: 4096 00000080:00200000:1.0:1789760787.906400:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79577088, bytes: 4096 00000080:00200000:1.0:1789760787.906409:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.906417:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.906419:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79577088, 79581184) 0 0 00000080:00200000:1.0:1789760787.906423:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.906434:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19428, 19428] 00020000:00200000:1.0:1789760787.906441:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79577088 chunk: [79577088, 79581184) 79581184, 4096 00020000:00200000:1.0:1789760787.906445:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.906459:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19428, 19428] 00000008:00010000:1.0:1789760787.906472:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.906478:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.906486:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79577088, 79581184) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.906497:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79577088=0x4be4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.906547:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.906549:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79581184 00000020:00010000:1.0:1789760787.906556:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.906559:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.906561:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.906564:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.906567:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.906575:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.906580:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.906595:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760787.906694:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 79577088<>79581184 00000080:00200000:1.0:1789760787.930023:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19428, 19428] 00000080:00200000:1.0:1789760787.930049:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.930054:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.930065:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79581184 i_blocks 155456 00000020:00200000:1.0:1789760787.930069:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.930076:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.930101:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79581184, count: 4096 00000080:00200000:1.0:1789760787.930107:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79581184, bytes: 4096 00000080:00200000:1.0:1789760787.930115:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.930122:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.930123:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79581184, 79585280) 0 0 00000080:00200000:1.0:1789760787.930127:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.930139:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19429, 19429] 00020000:00200000:1.0:1789760787.930145:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79581184 chunk: [79581184, 79585280) 79585280, 4096 00020000:00200000:1.0:1789760787.930149:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.930164:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19429, 19429] 00000008:00010000:1.0:1789760787.930178:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.930183:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.930192:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79581184, 79585280) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.930202:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79581184=0x4be5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.930247:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.930250:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79585280 00000020:00010000:1.0:1789760787.930256:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.930259:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.930262:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.930265:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.930268:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.930274:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.930279:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.930291:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760787.930376:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 79581184<>79585280 00000080:00200000:1.0:1789760787.942504:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19429, 19429] 00000080:00200000:1.0:1789760787.942517:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.942522:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.942533:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79585280 i_blocks 155464 00000020:00200000:1.0:1789760787.942536:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.942544:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.942567:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79585280, count: 4096 00000080:00200000:1.0:1789760787.942572:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79585280, bytes: 4096 00000080:00200000:1.0:1789760787.942579:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.942586:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.942598:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79585280, 79589376) 0 0 00000080:00200000:1.0:1789760787.942602:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.942614:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19430, 19430] 00020000:00200000:1.0:1789760787.942628:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79585280 chunk: [79585280, 79589376) 79589376, 4096 00020000:00200000:1.0:1789760787.942632:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.942645:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19430, 19430] 00000008:00010000:1.0:1789760787.942664:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.942668:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.942677:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79585280, 79589376) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.942686:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79585280=0x4be6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.942729:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.942732:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79589376 00000020:00010000:1.0:1789760787.942738:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.942741:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.942744:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.942747:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.942749:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.942756:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.942762:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.942775:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760787.942848:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 79585280<>79589376 00000080:00200000:1.0:1789760787.962960:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19430, 19430] 00000080:00200000:1.0:1789760787.962973:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.962978:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.962990:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79589376 i_blocks 155472 00000020:00200000:1.0:1789760787.962994:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.963002:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.963028:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79589376, count: 4096 00000080:00200000:1.0:1789760787.963061:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79589376, bytes: 4096 00000080:00200000:1.0:1789760787.963069:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.963076:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.963078:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79589376, 79593472) 0 0 00000080:00200000:1.0:1789760787.963082:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.963093:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19431, 19431] 00020000:00200000:1.0:1789760787.963100:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79589376 chunk: [79589376, 79593472) 79593472, 4096 00020000:00200000:1.0:1789760787.963104:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.963117:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19431, 19431] 00000008:00010000:1.0:1789760787.963129:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.963133:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.963141:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79589376, 79593472) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.963151:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79589376=0x4be7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.963193:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.963195:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79593472 00000020:00010000:1.0:1789760787.963201:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.963204:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.963207:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.963211:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.963213:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.963219:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.963225:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.963237:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760787.963308:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 79589376<>79593472 00000080:00200000:1.0:1789760787.973431:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19431, 19431] 00000080:00200000:1.0:1789760787.973443:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.973448:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.973459:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79593472 i_blocks 155480 00000020:00200000:1.0:1789760787.973462:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.973469:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.973497:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79593472, count: 4096 00000080:00200000:1.0:1789760787.973503:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79593472, bytes: 4096 00000080:00200000:1.0:1789760787.973510:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.973517:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.973519:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79593472, 79597568) 0 0 00000080:00200000:1.0:1789760787.973523:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.973534:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19432, 19432] 00020000:00200000:1.0:1789760787.973540:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79593472 chunk: [79593472, 79597568) 79597568, 4096 00020000:00200000:1.0:1789760787.973543:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.973557:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19432, 19432] 00000008:00010000:1.0:1789760787.973570:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.973575:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.973584:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79593472, 79597568) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.973594:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79593472=0x4be8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.973647:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.973650:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79597568 00000020:00010000:1.0:1789760787.973664:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.973667:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.973670:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.973673:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.973675:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.973681:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.973686:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.973700:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58cc0 00000008:00100000:1.0:1789760787.973774:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 79593472<>79597568 00000100:00100000:0.0:1789760787.983769:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503876864:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760787.983904:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19432, 19432] 00000080:00200000:1.0:1789760787.983916:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.983922:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.983934:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79597568 i_blocks 155488 00000020:00200000:1.0:1789760787.983937:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.983944:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.983970:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79597568, count: 4096 00000080:00200000:1.0:1789760787.983976:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79597568, bytes: 4096 00000080:00200000:1.0:1789760787.983984:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.983991:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.983993:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79597568, 79601664) 0 0 00000080:00200000:1.0:1789760787.983997:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.984009:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19433, 19433] 00020000:00200000:1.0:1789760787.984015:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79597568 chunk: [79597568, 79601664) 79601664, 4096 00020000:00200000:1.0:1789760787.984020:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.984057:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19433, 19433] 00000008:00010000:1.0:1789760787.984072:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.984077:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.984086:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79597568, 79601664) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.984096:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79597568=0x4be9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.984139:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.984141:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79601664 00000020:00010000:1.0:1789760787.984148:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.984151:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.984154:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.984156:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.984159:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.984164:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.984169:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.984182:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760787.984258:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 79597568<>79601664 00000100:00100000:0.0:1789760787.984311:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503876992:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760787.995201:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986740 gen 2 00000008:00010000:0.0:1789760787.995235:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58000 00000020:00200000:0.0:1789760787.995241:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760787.995250:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760787.995265:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58000 00000100:00100000:0.0:1789760787.995288:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503876992:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760787.995517:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19433, 19433] 00000080:00200000:1.0:1789760787.995529:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.995534:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.995545:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79601664 i_blocks 155496 00000020:00200000:1.0:1789760787.995548:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.995556:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.995582:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79601664, count: 4096 00000080:00200000:1.0:1789760787.995588:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79601664, bytes: 4096 00000080:00200000:1.0:1789760787.995595:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.995602:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760787.995604:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79601664, 79605760) 0 0 00000080:00200000:1.0:1789760787.995608:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.995634:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19434, 19434] 00020000:00200000:1.0:1789760787.995641:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79601664 chunk: [79601664, 79605760) 79605760, 4096 00020000:00200000:1.0:1789760787.995646:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.995826:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19434, 19434] 00000008:00010000:1.0:1789760787.995842:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.995847:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760787.995857:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79601664, 79605760) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.995867:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79601664=0x4bea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.995912:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.995914:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79605760 00000020:00010000:1.0:1789760787.995921:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.995925:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.995927:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.995930:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.995933:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.995939:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.995944:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.995957:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760787.996136:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 79601664<>79605760 00000100:00100000:0.0:1789760787.996184:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760788.000443:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503877120:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760788.009880:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986741 gen 2 00000008:00010000:0.0:1789760788.009923:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:0.0:1789760788.009929:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760788.009939:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760788.009961:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000100:00100000:0.0:1789760788.009987:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503877120:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.010560:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503877248:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.018427:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986742 gen 2 00000008:00010000:3.0:1789760788.018462:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760788.018468:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.018476:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.018494:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760788.018515:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503877248:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.019161:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503877376:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.039893:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986743 gen 2 00000008:00010000:3.0:1789760788.039930:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760788.039937:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.039945:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.039963:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760788.039986:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503877376:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.040596:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503877504:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.057465:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986744 gen 2 00000008:00010000:3.0:1789760788.057516:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760788.057523:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.057530:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.057548:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760788.057572:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503877504:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.058225:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760788.058239:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503877632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.068502:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986745 gen 2 00000008:00010000:3.0:1789760788.068536:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760788.068542:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.068550:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.068565:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760788.068585:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503877632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.069281:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760788.069296:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503877760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.081081:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986746 gen 2 00000008:00010000:3.0:1789760788.081119:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760788.081126:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.081133:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.081153:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760788.081179:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503877760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.081891:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760788.081906:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503877888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.091068:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986747 gen 2 00000008:00010000:3.0:1789760788.091103:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760788.091109:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.091118:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.091134:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760788.091157:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503877888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.091998:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760788.092013:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503878016:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.105968:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986748 gen 2 00000008:00010000:3.0:1789760788.106015:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760788.106022:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.106201:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.106223:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760788.106250:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503878016:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.107758:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760788.107784:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503878144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.115678:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986749 gen 2 00000008:00010000:3.0:1789760788.115794:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760788.115802:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.115813:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.115871:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760788.115902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503878144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.116605:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503878272:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.124448:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986750 gen 2 00000008:00010000:3.0:1789760788.124498:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760788.124506:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.124516:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.124539:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760788.124569:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503878272:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.126529:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503878400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.133963:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986751 gen 2 00000008:00010000:3.0:1789760788.134288:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760788.134304:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.134316:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.134339:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760788.134368:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503878400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.135743:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760788.135766:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503878528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.146128:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986752 gen 2 00000008:00010000:3.0:1789760788.146165:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760788.146172:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.146180:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.146203:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760788.146229:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503878528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.146969:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503878656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.155760:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986753 gen 2 00000008:00010000:3.0:1789760788.155798:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760788.155804:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.155814:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.155881:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760788.155905:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503878656:192.168.203.158@tcp:4:dd.0 00000080:00200008:0.0:1789760788.156215:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79654912, count: 4096 00000080:00200000:0.0:1789760788.156224:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79654912, bytes: 4096 00000080:00200000:0.0:1789760788.156233:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.156240:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.156241:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79654912, 79659008) 0 0 00000080:00200000:0.0:1789760788.156245:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.156257:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19447, 19447] 00020000:00200000:0.0:1789760788.156264:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79654912 chunk: [79654912, 79659008) 79659008, 4096 00020000:00200000:0.0:1789760788.156268:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.156281:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19447, 19447] 00000008:00010000:0.0:1789760788.156294:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.156300:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.156309:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79654912, 79659008) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.156320:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79654912=0x4bf7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.156480:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.156484:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79659008 00000020:00010000:0.0:1789760788.156553:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.156558:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.156561:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.156564:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.156567:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.156574:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.156580:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.156594:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760788.156834:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99c00 - object 0x280000401:2 offset 79654912<>79659008 00000100:00100000:1.0:1789760788.156905:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503878784:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.164232:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986754 gen 2 00000008:00010000:1.0:1789760788.164265:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760788.164272:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.164280:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.164346:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760788.164369:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503878784:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.165098:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19447, 19447] 00000080:00200000:0.0:1789760788.165110:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.165116:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.165128:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79659008 i_blocks 155608 00000020:00200000:0.0:1789760788.165131:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.165138:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.165162:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79659008, count: 4096 00000080:00200000:0.0:1789760788.165167:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79659008, bytes: 4096 00000080:00200000:0.0:1789760788.165175:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.165182:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.165184:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79659008, 79663104) 0 0 00000080:00200000:0.0:1789760788.165188:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.165199:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19448, 19448] 00020000:00200000:0.0:1789760788.165206:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79659008 chunk: [79659008, 79663104) 79663104, 4096 00020000:00200000:0.0:1789760788.165210:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.165224:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19448, 19448] 00000008:00010000:0.0:1789760788.165237:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.165241:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.165249:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79659008, 79663104) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.165260:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79659008=0x4bf8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.165303:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.165306:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79663104 00000020:00010000:0.0:1789760788.165311:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.165313:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.165316:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.165318:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.165321:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.165402:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.165410:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.165428:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760788.165561:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98380 - object 0x280000401:2 offset 79659008<>79663104 00000100:00100000:1.0:1789760788.165628:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503878912:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.172590:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986755 gen 2 00000008:00010000:1.0:1789760788.172644:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760788.172651:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.172667:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.172685:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760788.172710:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503878912:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.172865:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19448, 19448] 00000080:00200000:0.0:1789760788.172878:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.172882:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.172894:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79663104 i_blocks 155616 00000020:00200000:0.0:1789760788.172898:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.172906:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.172939:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79663104, count: 4096 00000080:00200000:0.0:1789760788.172945:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79663104, bytes: 4096 00000080:00200000:0.0:1789760788.172953:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.172960:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.172962:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79663104, 79667200) 0 0 00000080:00200000:0.0:1789760788.172966:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.172978:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19449, 19449] 00020000:00200000:0.0:1789760788.172984:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79663104 chunk: [79663104, 79667200) 79667200, 4096 00020000:00200000:0.0:1789760788.172987:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.173002:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19449, 19449] 00000008:00010000:0.0:1789760788.173016:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.173020:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.173027:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79663104, 79667200) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.173071:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79663104=0x4bf9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.173114:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.173116:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79667200 00000020:00010000:0.0:1789760788.173123:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.173126:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.173129:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.173132:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.173135:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.173142:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.173152:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.173167:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760788.173238:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9ad80 - object 0x280000401:2 offset 79663104<>79667200 00000100:00100000:1.0:1789760788.173847:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503879040:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.181519:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986756 gen 2 00000008:00010000:1.0:1789760788.181555:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760788.181562:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.181571:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.181591:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760788.181629:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503879040:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.182006:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19449, 19449] 00000080:00200000:0.0:1789760788.182018:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.182022:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.182066:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79667200 i_blocks 155624 00000020:00200000:0.0:1789760788.182069:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.182077:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.182102:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79667200, count: 4096 00000080:00200000:0.0:1789760788.182108:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79667200, bytes: 4096 00000080:00200000:0.0:1789760788.182115:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.182121:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.182123:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79667200, 79671296) 0 0 00000080:00200000:0.0:1789760788.182127:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.182138:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19450, 19450] 00020000:00200000:0.0:1789760788.182144:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79667200 chunk: [79667200, 79671296) 79671296, 4096 00020000:00200000:0.0:1789760788.182147:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.182160:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19450, 19450] 00000008:00010000:0.0:1789760788.182173:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.182177:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.182185:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79667200, 79671296) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.182196:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79667200=0x4bfa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.182233:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.182235:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79671296 00000020:00010000:0.0:1789760788.182240:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.182243:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.182246:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.182249:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.182251:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.182257:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.182262:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.182273:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760788.182353:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b100 - object 0x280000401:2 offset 79667200<>79671296 00000100:00100000:1.0:1789760788.182479:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503879168:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.192026:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986757 gen 2 00000008:00010000:1.0:1789760788.192082:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760788.192088:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.192097:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.192114:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760788.192135:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503879168:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.192217:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19450, 19450] 00000080:00200000:0.0:1789760788.192229:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.192233:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.192244:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79671296 i_blocks 155632 00000020:00200000:0.0:1789760788.192247:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.192255:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.192278:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79671296, count: 4096 00000080:00200000:0.0:1789760788.192284:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79671296, bytes: 4096 00000080:00200000:0.0:1789760788.192291:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.192297:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.192299:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79671296, 79675392) 0 0 00000080:00200000:0.0:1789760788.192303:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.192314:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19451, 19451] 00020000:00200000:0.0:1789760788.192320:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79671296 chunk: [79671296, 79675392) 79675392, 4096 00020000:00200000:0.0:1789760788.192323:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.192337:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19451, 19451] 00000008:00010000:0.0:1789760788.192358:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.192363:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.192370:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79671296, 79675392) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.192380:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79671296=0x4bfb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.192417:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.192420:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79675392 00000020:00010000:0.0:1789760788.192425:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.192428:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.192432:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.192434:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.192437:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.192443:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.192447:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.192459:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760788.192534:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9aa00 - object 0x280000401:2 offset 79671296<>79675392 00000100:00100000:1.0:1789760788.192772:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503879296:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.199892:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986758 gen 2 00000008:00010000:1.0:1789760788.199929:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760788.199935:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.199944:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.199963:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760788.199987:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503879296:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.200091:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19451, 19451] 00000080:00200000:0.0:1789760788.200103:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.200109:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.200121:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79675392 i_blocks 155640 00000020:00200000:0.0:1789760788.200124:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.200132:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.200157:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79675392, count: 4096 00000080:00200000:0.0:1789760788.200164:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79675392, bytes: 4096 00000080:00200000:0.0:1789760788.200170:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.200177:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.200179:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79675392, 79679488) 0 0 00000080:00200000:0.0:1789760788.200182:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.200195:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19452, 19452] 00020000:00200000:0.0:1789760788.200201:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79675392 chunk: [79675392, 79679488) 79679488, 4096 00020000:00200000:0.0:1789760788.200205:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.200217:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19452, 19452] 00000008:00010000:0.0:1789760788.200229:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.200234:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.200241:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79675392, 79679488) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.200253:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79675392=0x4bfc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.200295:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.200298:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79679488 00000020:00010000:0.0:1789760788.200304:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.200307:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.200310:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.200313:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.200315:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.200323:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.200328:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.200341:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760788.200414:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b800 - object 0x280000401:2 offset 79675392<>79679488 00000100:00100000:1.0:1789760788.200823:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503879424:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.213831:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986759 gen 2 00000008:00010000:1.0:1789760788.213867:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760788.213874:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.213883:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.213959:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760788.213984:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503879424:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.214224:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19452, 19452] 00000080:00200000:0.0:1789760788.214240:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.214246:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.214258:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79679488 i_blocks 155648 00000020:00200000:0.0:1789760788.214260:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.214278:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.214304:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79679488, count: 4096 00000080:00200000:0.0:1789760788.214309:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79679488, bytes: 4096 00000080:00200000:0.0:1789760788.214319:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.214326:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.214328:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79679488, 79683584) 0 0 00000080:00200000:0.0:1789760788.214332:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.214355:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19453, 19453] 00020000:00200000:0.0:1789760788.214362:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79679488 chunk: [79679488, 79683584) 79683584, 4096 00020000:00200000:0.0:1789760788.214367:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.214381:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19453, 19453] 00000008:00010000:0.0:1789760788.214395:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.214400:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.214410:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79679488, 79683584) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.214422:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79679488=0x4bfd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.214472:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.214475:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79683584 00000020:00010000:0.0:1789760788.214482:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.214486:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.214488:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.214504:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.214507:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.214514:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.214520:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.214536:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760788.214629:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b480 - object 0x280000401:2 offset 79679488<>79683584 00000100:00100000:1.0:1789760788.214687:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503879552:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.231500:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986760 gen 2 00000008:00010000:1.0:1789760788.231531:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760788.231548:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.231556:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.231576:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760788.231599:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503879552:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.232092:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19453, 19453] 00000080:00200000:0.0:1789760788.232106:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.232110:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.232122:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79683584 i_blocks 155656 00000020:00200000:0.0:1789760788.232126:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.232133:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.232157:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79683584, count: 4096 00000080:00200000:0.0:1789760788.232163:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79683584, bytes: 4096 00000080:00200000:0.0:1789760788.232170:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.232176:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.232178:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79683584, 79687680) 0 0 00000080:00200000:0.0:1789760788.232182:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.232194:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19454, 19454] 00020000:00200000:0.0:1789760788.232201:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79683584 chunk: [79683584, 79687680) 79687680, 4096 00020000:00200000:0.0:1789760788.232204:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.232219:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19454, 19454] 00000008:00010000:0.0:1789760788.232232:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.232237:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.232245:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79683584, 79687680) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.232256:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79683584=0x4bfe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.232355:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.232358:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79687680 00000020:00010000:0.0:1789760788.232366:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.232369:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.232372:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.232375:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.232378:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.232385:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.232392:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.232407:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760788.232477:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a680 - object 0x280000401:2 offset 79683584<>79687680 00000100:00100000:1.0:1789760788.232573:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503879680:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.248921:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986761 gen 2 00000008:00010000:1.0:1789760788.248963:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760788.248970:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.248979:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.248999:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760788.249023:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503879680:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.249151:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19454, 19454] 00000080:00200000:0.0:1789760788.249163:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.249169:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.249181:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79687680 i_blocks 155664 00000020:00200000:0.0:1789760788.249185:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.249193:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.249218:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79687680, count: 4096 00000080:00200000:0.0:1789760788.249224:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79687680, bytes: 4096 00000080:00200000:0.0:1789760788.249232:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.249239:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.249241:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79687680, 79691776) 0 0 00000080:00200000:0.0:1789760788.249245:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.249258:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19455, 19455] 00020000:00200000:0.0:1789760788.249264:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79687680 chunk: [79687680, 79691776) 79691776, 4096 00020000:00200000:0.0:1789760788.249279:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.249294:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19455, 19455] 00000008:00010000:0.0:1789760788.249307:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.249312:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.249321:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79687680, 79691776) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.249332:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79687680=0x4bff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.249384:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.249387:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79691776 00000020:00010000:0.0:1789760788.249393:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.249396:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.249399:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.249402:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.249404:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.249411:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.249416:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.249429:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760788.249507:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98000 - object 0x280000401:2 offset 79687680<>79691776 00000100:00100000:1.0:1789760788.249917:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503879808:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.260144:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986762 gen 2 00000008:00010000:1.0:1789760788.260180:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760788.260186:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.260194:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.260204:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760788.260227:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503879808:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.262106:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19455, 19455] 00000080:00200000:0.0:1789760788.262124:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.262129:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.262144:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79691776 i_blocks 155672 00000020:00200000:0.0:1789760788.262148:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.262159:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.262185:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79691776, count: 4096 00000080:00200000:0.0:1789760788.262192:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79691776, bytes: 4096 00000080:00200000:0.0:1789760788.262203:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.262212:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.262214:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79691776, 79695872) 0 0 00000080:00200000:0.0:1789760788.262218:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.262230:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19456, 19456] 00020000:00200000:0.0:1789760788.262237:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79691776 chunk: [79691776, 79695872) 79695872, 4096 00020000:00200000:0.0:1789760788.262241:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.262256:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19456, 19456] 00000008:00010000:0.0:1789760788.262270:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.262276:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.262285:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79691776, 79695872) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.262298:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79691776=0x4c00000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.262375:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.262378:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79695872 00000020:00010000:0.0:1789760788.262385:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.262388:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.262390:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.262393:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.262396:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.262403:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.262408:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.262424:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760788.262516:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98a80 - object 0x280000401:2 offset 79691776<>79695872 00000100:00100000:1.0:1789760788.262591:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503879936:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.273953:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986763 gen 2 00000008:00010000:1.0:1789760788.273991:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760788.273998:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.274007:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.274030:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760788.274085:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503879936:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.274784:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19456, 19456] 00000080:00200000:0.0:1789760788.274799:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.274804:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.274816:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79695872 i_blocks 155680 00000020:00200000:0.0:1789760788.274819:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.274827:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.274853:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79695872, count: 4096 00000080:00200000:0.0:1789760788.274860:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79695872, bytes: 4096 00000080:00200000:0.0:1789760788.274869:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.274876:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.274878:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79695872, 79699968) 0 0 00000080:00200000:0.0:1789760788.274882:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.274894:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19457, 19457] 00020000:00200000:0.0:1789760788.274900:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79695872 chunk: [79695872, 79699968) 79699968, 4096 00020000:00200000:0.0:1789760788.274904:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.274917:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19457, 19457] 00000008:00010000:0.0:1789760788.274931:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.274935:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.274943:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79695872, 79699968) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.274959:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79695872=0x4c01000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.275007:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.275010:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79699968 00000020:00010000:0.0:1789760788.275017:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.275020:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.275023:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.275026:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.275028:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.275067:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.275074:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.275088:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760788.275166:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 79695872<>79699968 00000100:00100000:1.0:1789760788.275257:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880064:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.285988:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986764 gen 2 00000008:00010000:1.0:1789760788.286029:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760788.286069:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.286080:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.286089:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760788.286112:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880064:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.286259:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19457, 19457] 00000080:00200000:0.0:1789760788.286274:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.286279:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.286292:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79699968 i_blocks 155688 00000020:00200000:0.0:1789760788.286295:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.286303:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.286329:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79699968, count: 4096 00000080:00200000:0.0:1789760788.286335:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79699968, bytes: 4096 00000080:00200000:0.0:1789760788.286823:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.286831:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.286832:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79699968, 79704064) 0 0 00000080:00200000:0.0:1789760788.286837:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.286848:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19458, 19458] 00020000:00200000:0.0:1789760788.286855:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79699968 chunk: [79699968, 79704064) 79704064, 4096 00020000:00200000:0.0:1789760788.286859:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.286874:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19458, 19458] 00000008:00010000:0.0:1789760788.286887:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.286892:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.286901:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79699968, 79704064) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.286911:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79699968=0x4c02000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.286955:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.286957:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79704064 00000020:00010000:0.0:1789760788.286963:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.286966:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.286969:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.286971:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.286974:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.286981:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.286986:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.287000:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760788.287112:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99880 - object 0x280000401:2 offset 79699968<>79704064 00000100:00100000:1.0:1789760788.287877:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760788.287908:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880192:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.302586:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986765 gen 2 00000008:00010000:1.0:1789760788.302630:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760788.302637:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.302647:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.302766:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760788.302792:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880192:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.302884:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19458, 19458] 00000080:00200000:0.0:1789760788.302908:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.302913:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.302924:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79704064 i_blocks 155696 00000020:00200000:0.0:1789760788.302927:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.302935:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.302960:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79704064, count: 4096 00000080:00200000:0.0:1789760788.302965:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79704064, bytes: 4096 00000080:00200000:0.0:1789760788.302974:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.302981:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.302983:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79704064, 79708160) 0 0 00000080:00200000:0.0:1789760788.302987:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.302999:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19459, 19459] 00020000:00200000:0.0:1789760788.303005:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79704064 chunk: [79704064, 79708160) 79708160, 4096 00020000:00200000:0.0:1789760788.303009:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.303024:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19459, 19459] 00000008:00010000:0.0:1789760788.303064:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.303069:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.303077:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79704064, 79708160) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.303088:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79704064=0x4c03000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.303133:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.303136:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79708160 00000020:00010000:0.0:1789760788.303143:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.303146:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.303148:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.303151:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.303154:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.303161:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.303167:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.303180:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760788.303252:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99f80 - object 0x280000401:2 offset 79704064<>79708160 00000100:00100000:1.0:1789760788.303587:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880320:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.315272:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986766 gen 2 00000008:00010000:1.0:1789760788.315309:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760788.315315:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.315325:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.315504:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760788.315531:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880320:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.316355:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19459, 19459] 00000080:00200000:0.0:1789760788.316370:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.316375:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.316387:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79708160 i_blocks 155704 00000020:00200000:0.0:1789760788.316391:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.316400:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.316425:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79708160, count: 4096 00000080:00200000:0.0:1789760788.316430:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79708160, bytes: 4096 00000080:00200000:0.0:1789760788.316439:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.316446:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.316447:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79708160, 79712256) 0 0 00000080:00200000:0.0:1789760788.316451:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.316465:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19460, 19460] 00020000:00200000:0.0:1789760788.316472:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79708160 chunk: [79708160, 79712256) 79712256, 4096 00020000:00200000:0.0:1789760788.316476:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.316503:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19460, 19460] 00000008:00010000:0.0:1789760788.316517:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.316521:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.316530:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79708160, 79712256) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.316541:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79708160=0x4c04000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.316584:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.316587:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79712256 00000020:00010000:0.0:1789760788.316594:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.316597:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.317415:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.317422:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.317424:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.317434:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.317442:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.317468:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760788.317768:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9bb80 - object 0x280000401:2 offset 79708160<>79712256 00000100:00100000:1.0:1789760788.317843:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503880448:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.332117:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986767 gen 2 00000008:00010000:1.0:1789760788.332157:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760788.332163:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.332172:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.332193:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760788.332217:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503880448:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.332314:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19460, 19460] 00000080:00200000:0.0:1789760788.332327:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.332332:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.332455:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79712256 i_blocks 155712 00000020:00200000:0.0:1789760788.332459:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.332468:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.332495:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79712256, count: 4096 00000080:00200000:0.0:1789760788.332502:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79712256, bytes: 4096 00000080:00200000:0.0:1789760788.332509:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.332515:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.332517:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79712256, 79716352) 0 0 00000080:00200000:0.0:1789760788.332521:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.332534:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19461, 19461] 00020000:00200000:0.0:1789760788.332542:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79712256 chunk: [79712256, 79716352) 79716352, 4096 00020000:00200000:0.0:1789760788.332545:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.332559:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19461, 19461] 00000008:00010000:0.0:1789760788.332572:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.332577:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.332585:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79712256, 79716352) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.332596:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79712256=0x4c05000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.332794:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.332796:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79716352 00000020:00010000:0.0:1789760788.332803:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.332807:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.332810:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.332813:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.332815:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.332822:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.332828:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.332842:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760788.332917:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98700 - object 0x280000401:2 offset 79712256<>79716352 00000100:00100000:1.0:1789760788.333478:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880576:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.350352:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986768 gen 2 00000008:00010000:1.0:1789760788.350387:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760788.350394:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.350403:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.350421:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760788.350446:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880576:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.350829:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19461, 19461] 00000080:00200000:0.0:1789760788.350842:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.350847:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.350859:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79716352 i_blocks 155720 00000020:00200000:0.0:1789760788.350862:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.350870:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.350895:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79716352, count: 4096 00000080:00200000:0.0:1789760788.350902:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79716352, bytes: 4096 00000080:00200000:0.0:1789760788.350909:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.350916:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.350918:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79716352, 79720448) 0 0 00000080:00200000:0.0:1789760788.350921:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.350933:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19462, 19462] 00020000:00200000:0.0:1789760788.350940:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79716352 chunk: [79716352, 79720448) 79720448, 4096 00020000:00200000:0.0:1789760788.350943:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.350959:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19462, 19462] 00000008:00010000:0.0:1789760788.350971:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.350976:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.350985:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79716352, 79720448) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.350995:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79716352=0x4c06000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.351072:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.351075:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79720448 00000020:00010000:0.0:1789760788.351081:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.351085:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.351088:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.351090:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.351093:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.351101:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.351106:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.351121:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760788.351190:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98700 - object 0x280000401:2 offset 79716352<>79720448 00000100:00100000:1.0:1789760788.351474:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760788.351489:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880704:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.364153:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986769 gen 2 00000008:00010000:1.0:1789760788.364189:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760788.364195:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.364203:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.364222:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760788.364247:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880704:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.365471:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19462, 19462] 00000080:00200000:0.0:1789760788.365488:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.365493:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.365506:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79720448 i_blocks 155728 00000020:00200000:0.0:1789760788.365511:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.365522:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.365547:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79720448, count: 4096 00000080:00200000:0.0:1789760788.365554:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79720448, bytes: 4096 00000080:00200000:0.0:1789760788.365563:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.365570:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.365572:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79720448, 79724544) 0 0 00000080:00200000:0.0:1789760788.365576:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.365587:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19463, 19463] 00020000:00200000:0.0:1789760788.365594:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79720448 chunk: [79720448, 79724544) 79724544, 4096 00020000:00200000:0.0:1789760788.365597:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.365612:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19463, 19463] 00000008:00010000:0.0:1789760788.365634:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.365639:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.365647:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79720448, 79724544) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.365670:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79720448=0x4c07000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.365718:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.365721:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79724544 00000020:00010000:0.0:1789760788.365727:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.365730:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.365733:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.365737:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.365739:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.365746:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.365752:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.365767:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760788.365844:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9bb80 - object 0x280000401:2 offset 79720448<>79724544 00000100:00100000:1.0:1789760788.366531:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880832:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.381817:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986770 gen 2 00000008:00010000:1.0:1789760788.381852:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760788.381859:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.381868:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.381887:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760788.381912:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503880832:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.382004:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19463, 19463] 00000080:00200000:0.0:1789760788.382017:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.382022:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.382069:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79724544 i_blocks 155736 00000020:00200000:0.0:1789760788.382073:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.382081:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.382107:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79724544, count: 4096 00000080:00200000:0.0:1789760788.382114:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79724544, bytes: 4096 00000080:00200000:0.0:1789760788.382121:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.382128:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.382130:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79724544, 79728640) 0 0 00000080:00200000:0.0:1789760788.382134:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.382145:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19464, 19464] 00020000:00200000:0.0:1789760788.382152:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79724544 chunk: [79724544, 79728640) 79728640, 4096 00020000:00200000:0.0:1789760788.382156:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.382170:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19464, 19464] 00000008:00010000:0.0:1789760788.382183:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.382187:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.382195:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79724544, 79728640) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.382206:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79724544=0x4c08000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.382249:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.382252:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79728640 00000020:00010000:0.0:1789760788.382258:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.382261:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.382263:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.382266:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.382268:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.382275:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.382281:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.382293:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760788.382797:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99f80 - object 0x280000401:2 offset 79724544<>79728640 00000100:00100000:1.0:1789760788.382848:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503880960:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.393814:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986771 gen 2 00000008:00010000:1.0:1789760788.393851:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760788.393858:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.393868:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.393892:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760788.393981:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503880960:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.394247:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19464, 19464] 00000080:00200000:0.0:1789760788.394260:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.394265:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.394277:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79728640 i_blocks 155744 00000020:00200000:0.0:1789760788.394280:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.394288:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.394314:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79728640, count: 4096 00000080:00200000:0.0:1789760788.394320:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79728640, bytes: 4096 00000080:00200000:0.0:1789760788.394328:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.394403:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.394407:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79728640, 79732736) 0 0 00000080:00200000:0.0:1789760788.394411:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.394424:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19465, 19465] 00020000:00200000:0.0:1789760788.394431:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79728640 chunk: [79728640, 79732736) 79732736, 4096 00020000:00200000:0.0:1789760788.394434:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.394449:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19465, 19465] 00000008:00010000:0.0:1789760788.394462:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.394466:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.394474:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79728640, 79732736) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.394484:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79728640=0x4c09000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.395004:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.395007:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79732736 00000020:00010000:0.0:1789760788.395015:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.395018:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.395021:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.395024:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.395027:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.395070:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.395077:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.395090:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760788.395164:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99880 - object 0x280000401:2 offset 79728640<>79732736 00000100:00100000:1.0:1789760788.395445:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503881088:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.411143:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986772 gen 2 00000008:00010000:1.0:1789760788.411184:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760788.411193:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.411203:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.411227:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760788.411257:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503881088:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.411583:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19465, 19465] 00000080:00200000:0.0:1789760788.411596:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.411602:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.411622:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79732736 i_blocks 155752 00000020:00200000:0.0:1789760788.411625:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.411634:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.411718:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79732736, count: 4096 00000080:00200000:0.0:1789760788.411725:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79732736, bytes: 4096 00000080:00200000:0.0:1789760788.411734:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.411740:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.411742:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79732736, 79736832) 0 0 00000080:00200000:0.0:1789760788.411747:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.411759:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19466, 19466] 00020000:00200000:0.0:1789760788.411767:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79732736 chunk: [79732736, 79736832) 79736832, 4096 00020000:00200000:0.0:1789760788.411772:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.411788:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19466, 19466] 00000008:00010000:0.0:1789760788.411804:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.411810:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.411820:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79732736, 79736832) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.411834:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79732736=0x4c0a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.411883:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.411885:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79736832 00000020:00010000:0.0:1789760788.411892:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.411895:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.411898:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.411901:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.411904:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.412026:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.412129:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.412149:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760788.412241:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 79732736<>79736832 00000100:00100000:1.0:1789760788.412302:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503881216:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.429340:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986773 gen 2 00000008:00010000:1.0:1789760788.429398:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760788.429405:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.429416:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.429445:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760788.429480:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503881216:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.429665:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19466, 19466] 00000080:00200000:0.0:1789760788.429678:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.429683:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.429696:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79736832 i_blocks 155760 00000020:00200000:0.0:1789760788.429700:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.429708:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.429734:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79736832, count: 4096 00000080:00200000:0.0:1789760788.429740:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79736832, bytes: 4096 00000080:00200000:0.0:1789760788.429748:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.429755:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.429757:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79736832, 79740928) 0 0 00000080:00200000:0.0:1789760788.429761:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.429775:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19467, 19467] 00020000:00200000:0.0:1789760788.429784:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79736832 chunk: [79736832, 79740928) 79740928, 4096 00020000:00200000:0.0:1789760788.429789:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.429805:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19467, 19467] 00000008:00010000:0.0:1789760788.429822:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.429828:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.429838:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79736832, 79740928) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.429851:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79736832=0x4c0b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.429908:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.429911:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79740928 00000020:00010000:0.0:1789760788.430024:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.430027:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.430031:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.430080:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.430085:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.430094:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.430100:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.430120:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760788.430189:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98a80 - object 0x280000401:2 offset 79736832<>79740928 00000100:00100000:1.0:1789760788.430281:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503881344:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.442869:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986774 gen 2 00000008:00010000:1.0:1789760788.442907:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760788.442913:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.442922:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.443375:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760788.443402:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503881344:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.444494:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19467, 19467] 00000080:00200000:0.0:1789760788.444511:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.444516:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.445990:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79740928 i_blocks 155768 00000020:00200000:0.0:1789760788.446011:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.446025:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.446114:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79740928, count: 4096 00000080:00200000:0.0:1789760788.446122:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79740928, bytes: 4096 00000080:00200000:0.0:1789760788.446459:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.446714:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.446717:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79740928, 79745024) 0 0 00000080:00200000:0.0:1789760788.446722:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.446737:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19468, 19468] 00020000:00200000:0.0:1789760788.446745:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79740928 chunk: [79740928, 79745024) 79745024, 4096 00020000:00200000:0.0:1789760788.446749:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.446766:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19468, 19468] 00000008:00010000:0.0:1789760788.446783:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.446788:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.446797:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79740928, 79745024) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.446810:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79740928=0x4c0c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.446862:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.446865:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79745024 00000020:00010000:0.0:1789760788.446873:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.446875:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.446878:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.446881:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.446883:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.446892:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.446897:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.446918:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760788.447005:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98000 - object 0x280000401:2 offset 79740928<>79745024 00000100:00100000:1.0:1789760788.447554:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760788.447577:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503881472:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.457846:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986775 gen 2 00000008:00010000:1.0:1789760788.457885:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760788.457891:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.457900:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.457927:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760788.457951:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503881472:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.458242:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19468, 19468] 00000080:00200000:0.0:1789760788.458254:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.458259:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.458270:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79745024 i_blocks 155776 00000020:00200000:0.0:1789760788.458274:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.458283:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.458310:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79745024, count: 4096 00000080:00200000:0.0:1789760788.458315:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79745024, bytes: 4096 00000080:00200000:0.0:1789760788.458323:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.458329:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.458331:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79745024, 79749120) 0 0 00000080:00200000:0.0:1789760788.458334:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.458404:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19469, 19469] 00020000:00200000:0.0:1789760788.458411:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79745024 chunk: [79745024, 79749120) 79749120, 4096 00020000:00200000:0.0:1789760788.458415:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.458429:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19469, 19469] 00000008:00010000:0.0:1789760788.458442:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.458446:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.458455:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79745024, 79749120) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.458465:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79745024=0x4c0d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.458508:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.458511:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79749120 00000020:00010000:0.0:1789760788.458518:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.458521:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.458524:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.458526:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.458529:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.458537:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.458542:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.458557:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760788.458638:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a680 - object 0x280000401:2 offset 79745024<>79749120 00000100:00100000:1.0:1789760788.462462:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760788.462488:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503881600:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.472576:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986776 gen 2 00000008:00010000:1.0:1789760788.472621:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760788.472628:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.472637:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.472664:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760788.472690:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503881600:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760788.472823:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19469, 19469] 00000080:00200000:0.0:1789760788.472837:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.472842:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.472854:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79749120 i_blocks 155784 00000020:00200000:0.0:1789760788.472857:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.472865:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.472890:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79749120, count: 4096 00000080:00200000:0.0:1789760788.472895:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79749120, bytes: 4096 00000080:00200000:0.0:1789760788.472903:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.472910:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760788.472912:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79749120, 79753216) 0 0 00000080:00200000:0.0:1789760788.472916:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.472927:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19470, 19470] 00020000:00200000:0.0:1789760788.472934:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79749120 chunk: [79749120, 79753216) 79753216, 4096 00020000:00200000:0.0:1789760788.472938:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.472952:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19470, 19470] 00000008:00010000:0.0:1789760788.472965:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.472969:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760788.472979:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79749120, 79753216) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.472989:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79749120=0x4c0e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.473030:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.473138:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79753216 00000020:00010000:0.0:1789760788.473147:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.473150:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.473153:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.473156:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.473159:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.473167:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.473172:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.473187:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760788.473264:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b480 - object 0x280000401:2 offset 79749120<>79753216 00000100:00100000:1.0:1789760788.473316:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503881728:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.481766:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986777 gen 2 00000008:00010000:1.0:1789760788.481804:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760788.481810:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.481818:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.481837:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760788.481861:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503881728:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.482515:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503881856:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.490282:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986778 gen 2 00000008:00010000:3.0:1789760788.490317:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760788.490324:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.490332:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.490356:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760788.490382:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503881856:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.490951:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503881984:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.498391:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986779 gen 2 00000008:00010000:3.0:1789760788.498426:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760788.498433:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.498443:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.498461:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760788.498484:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503881984:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.498997:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503882112:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.505900:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986780 gen 2 00000008:00010000:3.0:1789760788.505934:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760788.505941:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.505949:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.505967:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760788.505988:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503882112:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.507470:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503882240:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.520210:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986781 gen 2 00000008:00010000:3.0:1789760788.520245:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760788.520252:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.520260:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.520289:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760788.520311:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503882240:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.521200:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503882368:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.536808:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986782 gen 2 00000008:00010000:3.0:1789760788.536855:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760788.536861:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.536869:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.536895:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760788.536920:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503882368:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.538168:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503882496:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.559729:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986783 gen 2 00000008:00010000:3.0:1789760788.559765:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760788.559773:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.559781:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.559846:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760788.559869:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503882496:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.560840:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503882624:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.575565:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986784 gen 2 00000008:00010000:3.0:1789760788.575599:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760788.575605:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.575624:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.575642:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760788.575666:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503882624:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760788.576134:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c480 00000008:00100000:3.0:1789760788.576209:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79781888<>79785984 00000100:00100000:3.0:1789760788.576402:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503882752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.586223:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986785 gen 2 00000008:00010000:3.0:1789760788.586274:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760788.586281:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.586290:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.586307:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760788.586332:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503882752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.587030:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503882880:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.600071:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986786 gen 2 00000008:00010000:3.0:1789760788.600110:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760788.600116:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.600124:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.600142:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760788.600166:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503882880:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.600933:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503883008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.609375:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986787 gen 2 00000008:00010000:3.0:1789760788.609413:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760788.609420:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.609428:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.609442:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760788.609464:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503883008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.610021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503883136:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.618311:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986788 gen 2 00000008:00010000:3.0:1789760788.618351:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760788.618358:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.618367:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.618387:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760788.618411:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503883136:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.619130:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503883264:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.626198:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986789 gen 2 00000008:00010000:3.0:1789760788.626234:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760788.626241:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.626249:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.626276:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760788.626300:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503883264:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.627360:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503883392:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.637271:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986790 gen 2 00000008:00010000:3.0:1789760788.637309:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760788.637316:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.637324:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.637342:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760788.637366:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503883392:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.637932:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503883520:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.646802:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986791 gen 2 00000008:00010000:3.0:1789760788.646840:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760788.646847:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.646856:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.646874:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760788.646896:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503883520:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.649584:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760788.649603:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503883648:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.664091:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986792 gen 2 00000008:00010000:3.0:1789760788.664123:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760788.664129:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.664137:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.664153:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760788.664175:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503883648:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.676307:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986793 gen 2 00000008:00010000:3.0:1789760788.676351:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760788.676358:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.676367:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.676383:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760788.676407:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503883776:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.677006:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760788.677020:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503883904:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.689374:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986794 gen 2 00000008:00010000:3.0:1789760788.689409:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760788.689417:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.689424:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.689444:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760788.689467:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503883904:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.690139:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760788.690153:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503884032:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.698323:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986795 gen 2 00000008:00010000:3.0:1789760788.698369:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760788.698375:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.698384:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.698402:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760788.698426:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503884032:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.700938:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503884160:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.712202:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986796 gen 2 00000008:00010000:3.0:1789760788.712237:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760788.712243:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.712250:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.712269:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760788.712291:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503884160:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.713102:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503884288:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.727141:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986797 gen 2 00000008:00010000:3.0:1789760788.727178:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760788.727184:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.727193:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.727212:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760788.727235:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503884288:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.727896:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760788.727909:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503884416:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.736277:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986798 gen 2 00000008:00010000:3.0:1789760788.736315:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760788.736321:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.736329:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.736437:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760788.736462:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503884416:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.737158:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503884544:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.745975:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986799 gen 2 00000008:00010000:3.0:1789760788.746011:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760788.746018:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.746027:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.746081:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760788.746104:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503884544:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.746640:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503884672:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.757342:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986800 gen 2 00000008:00010000:3.0:1789760788.757377:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760788.757384:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.757393:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.757412:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760788.757437:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503884672:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.759752:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760788.759774:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503884800:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.769232:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986801 gen 2 00000008:00010000:3.0:1789760788.769263:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760788.769269:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.769277:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.769323:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760788.769347:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503884800:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.776363:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760788.776391:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503884928:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.791314:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986802 gen 2 00000008:00010000:3.0:1789760788.791360:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760788.791366:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.791375:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.791396:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760788.791418:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503884928:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.792642:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503885056:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.803368:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986803 gen 2 00000008:00010000:3.0:1789760788.803404:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760788.803410:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.803418:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.803475:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760788.803513:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503885056:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.804151:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885184:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.815232:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986804 gen 2 00000008:00010000:3.0:1789760788.815345:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760788.815351:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.815359:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.815378:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760788.815402:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885184:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.816016:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503885312:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.831257:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986805 gen 2 00000008:00010000:3.0:1789760788.831293:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760788.831299:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.831307:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.831324:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760788.831353:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503885312:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.831987:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885440:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.861708:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986806 gen 2 00000008:00010000:3.0:1789760788.861745:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760788.861752:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.861761:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.861779:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760788.861803:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885440:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.862596:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760788.862624:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885568:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.875351:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986807 gen 2 00000008:00010000:3.0:1789760788.875388:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760788.875395:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.875413:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.875431:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760788.875456:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885568:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.876211:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885696:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.895714:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986808 gen 2 00000008:00010000:3.0:1789760788.895752:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760788.895759:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.895769:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.895792:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760788.895820:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885696:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.896834:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760788.896852:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885824:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.914152:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986809 gen 2 00000008:00010000:3.0:1789760788.914186:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760788.914193:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.914202:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.914221:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760788.914245:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885824:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.915503:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885952:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.929295:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986810 gen 2 00000008:00010000:3.0:1789760788.929329:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760788.929334:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.929352:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.929370:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760788.929391:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503885952:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760788.931763:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19503, 19503] 00000080:00200000:1.0:1789760788.931780:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.931786:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.931799:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79888384 i_blocks 156056 00000020:00200000:1.0:1789760788.931803:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.931814:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.931839:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79888384, count: 4096 00000080:00200000:1.0:1789760788.931846:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79888384, bytes: 4096 00000080:00200000:1.0:1789760788.931857:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.931864:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760788.931866:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79888384, 79892480) 0 0 00000080:00200000:1.0:1789760788.931870:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.931881:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19504, 19504] 00020000:00200000:1.0:1789760788.931888:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79888384 chunk: [79888384, 79892480) 79892480, 4096 00020000:00200000:1.0:1789760788.931891:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.931907:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19504, 19504] 00000008:00010000:1.0:1789760788.931920:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.931925:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760788.931934:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79888384, 79892480) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.931944:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79888384=0x4c30000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.931991:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.931994:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79892480 00000020:00010000:1.0:1789760788.932000:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.932003:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.932006:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.932009:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418e48 (ffff9018c668f1b8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.932012:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018c668f1b8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.932018:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.932023:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.932072:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760788.932150:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 79888384<>79892480 00000100:00100000:1.0:1789760788.932181:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760788.932201:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503886080:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760788.944518:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986811 gen 2 00000008:00010000:1.0:1789760788.944555:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:1.0:1789760788.944561:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.944570:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.944592:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:1.0:1789760788.944627:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503886080:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760788.945076:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19504, 19504] 00000080:00200000:3.0:1789760788.945090:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760788.945095:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760788.945108:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79892480 i_blocks 156064 00000020:00200000:3.0:1789760788.945195:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760788.945205:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760788.945255:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79892480, count: 4096 00000080:00200000:3.0:1789760788.945263:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79892480, bytes: 4096 00000080:00200000:3.0:1789760788.945272:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760788.945280:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760788.945282:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79892480, 79896576) 0 0 00000080:00200000:3.0:1789760788.945286:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760788.945300:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19505, 19505] 00020000:00200000:3.0:1789760788.945306:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79892480 chunk: [79892480, 79896576) 79896576, 4096 00020000:00200000:3.0:1789760788.945310:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760788.945325:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19505, 19505] 00000008:00010000:3.0:1789760788.945338:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760788.945343:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760788.945352:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79892480, 79896576) append 0 append lockpos 0 00000080:00200000:3.0:1789760788.945362:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79892480=0x4c31000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760788.945406:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760788.945408:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79896576 00000020:00010000:3.0:1789760788.945415:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760788.945419:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.945422:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.945426:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760788.945428:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760788.945434:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760788.945439:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760788.945455:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760788.945543:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79892480<>79896576 00000100:00100000:3.0:1789760788.945579:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760788.945592:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503886208:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760788.958849:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986812 gen 2 00000008:00010000:3.0:1789760788.958886:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760788.958891:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.958900:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.958922:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760788.958946:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503886208:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760788.959364:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19505, 19505] 00000080:00200000:1.0:1789760788.959376:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.959381:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.959394:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79896576 i_blocks 156072 00000020:00200000:1.0:1789760788.959397:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.959404:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.959429:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79896576, count: 4096 00000080:00200000:1.0:1789760788.959435:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79896576, bytes: 4096 00000080:00200000:1.0:1789760788.959442:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.959448:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760788.959450:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79896576, 79900672) 0 0 00000080:00200000:1.0:1789760788.959454:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.959467:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19506, 19506] 00020000:00200000:1.0:1789760788.959474:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79896576 chunk: [79896576, 79900672) 79900672, 4096 00020000:00200000:1.0:1789760788.959477:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.959499:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19506, 19506] 00000008:00010000:1.0:1789760788.959513:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.959518:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760788.959527:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79896576, 79900672) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.959538:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79896576=0x4c32000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.959580:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.959583:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79900672 00000020:00010000:1.0:1789760788.959589:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.959593:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.959596:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.959599:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.959601:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.959607:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.959623:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.959637:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760788.959716:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 79896576<>79900672 00000100:00100000:0.0:1789760788.959932:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760788.959948:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503886336:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760788.971001:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986813 gen 2 00000008:00010000:0.0:1789760788.971067:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:0.0:1789760788.971074:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760788.971082:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760788.971103:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:0.0:1789760788.971127:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503886336:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760788.971256:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19506, 19506] 00000080:00200000:1.0:1789760788.971267:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.971272:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.971285:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79900672 i_blocks 156080 00000020:00200000:1.0:1789760788.971289:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.971296:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.971320:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79900672, count: 4096 00000080:00200000:1.0:1789760788.971326:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79900672, bytes: 4096 00000080:00200000:1.0:1789760788.971333:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.971340:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760788.971342:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79900672, 79904768) 0 0 00000080:00200000:1.0:1789760788.971346:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.971358:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19507, 19507] 00020000:00200000:1.0:1789760788.971364:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79900672 chunk: [79900672, 79904768) 79904768, 4096 00020000:00200000:1.0:1789760788.971368:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.971383:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19507, 19507] 00000008:00010000:1.0:1789760788.971396:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.971401:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760788.971410:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79900672, 79904768) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.971420:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79900672=0x4c33000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.971465:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.971468:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79904768 00000020:00010000:1.0:1789760788.971475:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.971479:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.971482:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.971485:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.971488:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.971494:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.971500:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.971515:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760788.971588:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 79900672<>79904768 00000100:00100000:0.0:1789760788.971724:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760788.971738:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503886464:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760788.981633:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986814 gen 2 00000008:00010000:0.0:1789760788.981676:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760788.981683:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760788.981692:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760788.981713:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760788.981738:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503886464:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760788.982541:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19507, 19507] 00000080:00200000:1.0:1789760788.982554:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.982560:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.982572:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79904768 i_blocks 156088 00000020:00200000:1.0:1789760788.982576:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.982586:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.982629:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79904768, count: 4096 00000080:00200000:1.0:1789760788.982637:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79904768, bytes: 4096 00000080:00200000:1.0:1789760788.982982:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.982991:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760788.982993:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79904768, 79908864) 0 0 00000080:00200000:1.0:1789760788.982997:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.983009:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19508, 19508] 00020000:00200000:1.0:1789760788.983016:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79904768 chunk: [79904768, 79908864) 79908864, 4096 00020000:00200000:1.0:1789760788.983020:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.983065:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19508, 19508] 00000008:00010000:1.0:1789760788.983080:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.983085:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760788.983094:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79904768, 79908864) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.983107:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79904768=0x4c34000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.983161:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.983164:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79908864 00000020:00010000:1.0:1789760788.983248:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.983252:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.983255:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.983258:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.983261:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.983710:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.983718:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.983735:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760788.983826:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 79904768<>79908864 00000100:00100000:0.0:1789760788.984206:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760788.984229:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503886592:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760788.993407:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986815 gen 2 00000008:00010000:0.0:1789760788.993451:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760788.993458:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760788.993469:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760788.993679:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:0.0:1789760788.993703:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503886592:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760788.997581:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19508, 19508] 00000080:00200000:1.0:1789760788.997600:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.997964:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.997979:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79908864 i_blocks 156096 00000020:00200000:1.0:1789760788.997984:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.997996:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.998023:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79908864, count: 4096 00000080:00200000:1.0:1789760788.998030:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79908864, bytes: 4096 00000080:00200000:1.0:1789760788.998057:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.998065:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760788.998068:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79908864, 79912960) 0 0 00000080:00200000:1.0:1789760788.998071:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.998083:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19509, 19509] 00020000:00200000:1.0:1789760788.998089:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79908864 chunk: [79908864, 79912960) 79912960, 4096 00020000:00200000:1.0:1789760788.998093:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.998107:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19509, 19509] 00000008:00010000:1.0:1789760788.998120:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.998125:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760788.998133:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79908864, 79912960) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.998143:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79908864=0x4c35000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.998309:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.998312:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79912960 00000020:00010000:1.0:1789760788.998318:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.998322:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.998325:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.998328:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.998330:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.998338:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.998406:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.998429:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760788.998583:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 79908864<>79912960 00000100:00100000:0.0:1789760788.998838:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503886720:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760789.008477:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986816 gen 2 00000008:00010000:0.0:1789760789.008525:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760789.008532:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.008541:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.008560:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000100:00100000:0.0:1789760789.008583:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503886720:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.018116:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986817 gen 2 00000008:00010000:3.0:1789760789.018156:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760789.018162:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.018172:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.018190:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760789.018214:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503886848:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.018758:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503886976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.040658:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986818 gen 2 00000008:00010000:3.0:1789760789.041786:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760789.041797:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.041806:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.041827:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760789.041853:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503886976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.042912:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887104:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.057016:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986819 gen 2 00000008:00010000:3.0:1789760789.057093:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760789.057100:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.057109:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.057127:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760789.057145:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887104:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.058226:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760789.058244:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.076096:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986820 gen 2 00000008:00010000:3.0:1789760789.076133:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760789.076139:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.076149:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.076169:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760789.076192:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.076983:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887360:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.090076:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986821 gen 2 00000008:00010000:3.0:1789760789.090114:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760789.090121:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.090129:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.090152:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760789.090175:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887360:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.090968:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760789.090983:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887488:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.105900:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986822 gen 2 00000008:00010000:3.0:1789760789.105947:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760789.105955:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.105964:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.105982:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760789.106005:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887488:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760789.106607:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c000 00000008:00100000:3.0:1789760789.106702:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79937536<>79941632 00000100:00100000:3.0:1789760789.106747:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887616:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.139820:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986823 gen 2 00000008:00010000:3.0:1789760789.139855:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760789.139863:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.139871:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.139893:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760789.139916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887616:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.140654:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760789.140669:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887744:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.150484:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986824 gen 2 00000008:00010000:3.0:1789760789.150530:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760789.150536:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.150545:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.150565:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760789.150591:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887744:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.151255:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887872:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.169686:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986825 gen 2 00000008:00010000:3.0:1789760789.169723:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760789.169729:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.169738:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.169870:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760789.169894:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503887872:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.171009:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503888000:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.184869:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986826 gen 2 00000008:00010000:3.0:1789760789.184904:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760789.184910:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.184919:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.184976:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760789.185008:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503888000:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.185930:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503888128:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.201237:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986827 gen 2 00000008:00010000:3.0:1789760789.201270:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760789.201276:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.201285:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.201340:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760789.201362:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503888128:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.201896:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503888256:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.223241:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986828 gen 2 00000008:00010000:3.0:1789760789.223278:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760789.223284:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.223293:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.223309:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760789.223334:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503888256:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.223833:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503888384:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.234857:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986829 gen 2 00000008:00010000:3.0:1789760789.234894:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760789.234901:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.234910:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.234930:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760789.235015:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503888384:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.235992:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503888512:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.250195:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986830 gen 2 00000008:00010000:3.0:1789760789.250299:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760789.250432:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.250441:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.250462:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760789.250486:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503888512:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.250579:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19523, 19523] 00000080:00200000:0.0:1789760789.250592:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.250597:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.250620:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79970304 i_blocks 156216 00000020:00200000:0.0:1789760789.250624:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.251393:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.251445:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79970304, count: 4096 00000080:00200000:0.0:1789760789.251452:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79970304, bytes: 4096 00000080:00200000:0.0:1789760789.251462:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.251469:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.251471:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79970304, 79974400) 0 0 00000080:00200000:0.0:1789760789.251476:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.251490:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19524, 19524] 00020000:00200000:0.0:1789760789.251497:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79970304 chunk: [79970304, 79974400) 79974400, 4096 00020000:00200000:0.0:1789760789.251572:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.251588:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19524, 19524] 00000008:00010000:0.0:1789760789.251803:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.251809:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.251820:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79970304, 79974400) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.251830:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79970304=0x4c44000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.251878:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.251882:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79974400 00000020:00010000:0.0:1789760789.251889:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.251893:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.251895:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.251899:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.251901:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.252766:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.252773:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.252792:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760789.252877:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b800 - object 0x280000401:2 offset 79970304<>79974400 00000100:00100000:0.0:1789760789.252924:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760789.252943:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503888640:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760789.264246:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986831 gen 2 00000008:00010000:0.0:1789760789.264283:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:0.0:1789760789.264291:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.264300:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.264319:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:0.0:1789760789.264345:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503888640:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760789.264357:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19524, 19524] 00000080:00200000:3.0:1789760789.264383:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.264389:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.264520:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 79974400 i_blocks 156224 00000020:00200000:3.0:1789760789.264525:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.264533:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.264560:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 79974400, count: 4096 00000080:00200000:3.0:1789760789.264566:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 79974400, bytes: 4096 00000080:00200000:3.0:1789760789.264574:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.264581:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760789.264583:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [79974400, 79978496) 0 0 00000080:00200000:3.0:1789760789.264587:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.264599:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19525, 19525] 00020000:00200000:3.0:1789760789.264616:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 79974400 chunk: [79974400, 79978496) 79978496, 4096 00020000:00200000:3.0:1789760789.264620:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.264635:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19525, 19525] 00000008:00010000:3.0:1789760789.264648:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.264653:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760789.264782:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [79974400, 79978496) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.264793:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=79974400=0x4c45000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.264834:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.264836:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 79978496 00000020:00010000:3.0:1789760789.264842:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.264844:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.264846:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.264848:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.264850:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.264854:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.264858:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.264870:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760789.264931:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79974400<>79978496 00000100:00100000:3.0:1789760789.264961:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760789.264971:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503888768:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.275519:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986832 gen 2 00000008:00010000:3.0:1789760789.275558:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760789.275564:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.275572:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.275590:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760789.275628:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503888768:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760789.276339:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000008:00100000:3.0:1789760789.276418:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 79978496<>79982592 00000100:00100000:3.0:1789760789.276457:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503888896:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.287653:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986833 gen 2 00000008:00010000:3.0:1789760789.287699:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760789.287705:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.287714:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.287734:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760789.287755:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503888896:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.288414:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889024:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.301502:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986834 gen 2 00000008:00010000:3.0:1789760789.301539:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760789.301545:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.301553:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.301570:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760789.301593:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889024:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.302382:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760789.302396:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889152:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.313666:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986835 gen 2 00000008:00010000:3.0:1789760789.313702:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760789.313708:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.313718:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.313737:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760789.313762:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889152:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.317448:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889280:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.338103:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986836 gen 2 00000008:00010000:3.0:1789760789.338139:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760789.338146:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.338154:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.338175:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760789.338200:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889280:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.338909:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760789.338923:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889408:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.357076:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986837 gen 2 00000008:00010000:3.0:1789760789.357112:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760789.357119:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.357128:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.357146:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760789.357171:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889408:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.357953:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889536:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.378676:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986838 gen 2 00000008:00010000:3.0:1789760789.378714:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760789.378721:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.378730:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.378748:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760789.378773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889536:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.379629:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503889664:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.395226:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986839 gen 2 00000008:00010000:3.0:1789760789.395262:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760789.395269:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.395278:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.395296:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760789.395320:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503889664:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.396789:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889792:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.422320:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986840 gen 2 00000008:00010000:3.0:1789760789.422356:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760789.422363:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.422371:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.422391:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760789.422424:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503889792:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.423074:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503889920:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.441135:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986841 gen 2 00000008:00010000:3.0:1789760789.441171:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760789.441177:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.441185:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.441204:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760789.441226:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503889920:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.442071:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503890048:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.460116:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986842 gen 2 00000008:00010000:3.0:1789760789.460155:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760789.460162:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.460170:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.460236:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760789.460263:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503890048:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.463815:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503890176:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.483387:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986843 gen 2 00000008:00010000:3.0:1789760789.483433:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760789.483440:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.483449:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.483472:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760789.483497:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503890176:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.485115:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19536, 19536] 00000080:00200000:0.0:1789760789.485136:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.485142:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.485156:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80023552 i_blocks 156320 00000020:00200000:0.0:1789760789.485160:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.485169:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.485206:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80023552, count: 4096 00000080:00200000:0.0:1789760789.485212:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80023552, bytes: 4096 00000080:00200000:0.0:1789760789.485221:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.485228:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.485230:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80023552, 80027648) 0 0 00000080:00200000:0.0:1789760789.485234:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.485246:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19537, 19537] 00020000:00200000:0.0:1789760789.485252:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80023552 chunk: [80023552, 80027648) 80027648, 4096 00020000:00200000:0.0:1789760789.485255:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.485270:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19537, 19537] 00000008:00010000:0.0:1789760789.485284:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.485288:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.485296:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80023552, 80027648) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.485306:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80023552=0x4c51000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.485351:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.485354:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80027648 00000020:00010000:0.0:1789760789.485360:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.485363:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.485366:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.485369:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.485372:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.485379:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.485384:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.485398:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760789.485473:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b800 - object 0x280000401:2 offset 80023552<>80027648 00000100:00100000:1.0:1789760789.485553:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503890304:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.499873:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986844 gen 2 00000008:00010000:1.0:1789760789.499908:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760789.499914:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.499923:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.499942:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760789.499966:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503890304:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.500500:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19537, 19537] 00000080:00200000:0.0:1789760789.500513:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.500518:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.500531:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80027648 i_blocks 156328 00000020:00200000:0.0:1789760789.500535:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.500543:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.500568:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80027648, count: 4096 00000080:00200000:0.0:1789760789.500575:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80027648, bytes: 4096 00000080:00200000:0.0:1789760789.500583:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.500591:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.500593:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80027648, 80031744) 0 0 00000080:00200000:0.0:1789760789.500597:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.500617:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19538, 19538] 00020000:00200000:0.0:1789760789.500625:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80027648 chunk: [80027648, 80031744) 80031744, 4096 00020000:00200000:0.0:1789760789.500630:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.500645:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19538, 19538] 00000008:00010000:0.0:1789760789.500658:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.500663:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.500671:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80027648, 80031744) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.500682:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80027648=0x4c52000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.500727:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.500731:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80031744 00000020:00010000:0.0:1789760789.500738:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.500741:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.500744:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.500747:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.500750:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.500757:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.500763:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.500778:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760789.500871:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9aa00 - object 0x280000401:2 offset 80027648<>80031744 00000100:00100000:1.0:1789760789.500949:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503890432:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.518487:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986845 gen 2 00000008:00010000:1.0:1789760789.518641:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760789.518709:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.518720:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.518741:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760789.518766:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503890432:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.518863:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19538, 19538] 00000080:00200000:0.0:1789760789.518874:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.518879:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.518891:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80031744 i_blocks 156336 00000020:00200000:0.0:1789760789.518894:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.518902:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.518926:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80031744, count: 4096 00000080:00200000:0.0:1789760789.518932:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80031744, bytes: 4096 00000080:00200000:0.0:1789760789.518941:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.518947:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.518949:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80031744, 80035840) 0 0 00000080:00200000:0.0:1789760789.518953:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.518965:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19539, 19539] 00020000:00200000:0.0:1789760789.518971:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80031744 chunk: [80031744, 80035840) 80035840, 4096 00020000:00200000:0.0:1789760789.518974:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.518989:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19539, 19539] 00000008:00010000:0.0:1789760789.519001:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.519005:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.519014:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80031744, 80035840) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.519024:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80031744=0x4c53000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.519087:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.519090:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80035840 00000020:00010000:0.0:1789760789.519097:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.519100:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.519103:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.519106:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.519109:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.519116:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.519122:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.519136:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760789.519455:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b100 - object 0x280000401:2 offset 80031744<>80035840 00000100:00100000:1.0:1789760789.519893:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503890560:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.534032:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986846 gen 2 00000008:00010000:1.0:1789760789.534327:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760789.534332:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.534342:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.534361:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000080:00200000:0.0:1789760789.534627:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19539, 19539] 00000080:00200000:0.0:1789760789.534641:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.534654:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.534668:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80035840 i_blocks 156344 00000020:00200000:0.0:1789760789.534671:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.534679:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.534704:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80035840, count: 4096 00000080:00200000:0.0:1789760789.534711:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80035840, bytes: 4096 00000080:00200000:0.0:1789760789.534719:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.534727:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.534729:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80035840, 80039936) 0 0 00000080:00200000:0.0:1789760789.534733:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.534746:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19540, 19540] 00020000:00200000:0.0:1789760789.534752:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80035840 chunk: [80035840, 80039936) 80039936, 4096 00020000:00200000:0.0:1789760789.534757:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.534771:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19540, 19540] 00000008:00010000:0.0:1789760789.534784:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.534789:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.534798:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80035840, 80039936) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.534808:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80035840=0x4c54000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.534845:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.534848:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80039936 00000020:00010000:0.0:1789760789.534854:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.534857:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.534861:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.534863:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.534866:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.534873:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.534879:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760789.535280:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503890560:192.168.203.158@tcp:4:dd.0 00000008:00010000:1.0:1789760789.535319:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:1.0:1789760789.535501:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 80035840<>80039936 00000100:00100000:1.0:1789760789.535536:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760789.535548:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503890688:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.553442:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986847 gen 2 00000008:00010000:1.0:1789760789.553479:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760789.553485:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.553493:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.553570:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760789.553598:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503890688:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.556779:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19540, 19540] 00000080:00200000:0.0:1789760789.556799:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.556805:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.556821:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80039936 i_blocks 156352 00000020:00200000:0.0:1789760789.556826:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.556838:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.556864:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80039936, count: 4096 00000080:00200000:0.0:1789760789.556871:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80039936, bytes: 4096 00000080:00200000:0.0:1789760789.556881:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.556889:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.556891:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80039936, 80044032) 0 0 00000080:00200000:0.0:1789760789.556894:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.556906:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19541, 19541] 00020000:00200000:0.0:1789760789.556923:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80039936 chunk: [80039936, 80044032) 80044032, 4096 00020000:00200000:0.0:1789760789.556927:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.556943:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19541, 19541] 00000008:00010000:0.0:1789760789.556957:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.556961:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.556970:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80039936, 80044032) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.556980:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80039936=0x4c55000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.557071:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.557074:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80044032 00000020:00010000:0.0:1789760789.557080:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.557083:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.557086:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.557089:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.557092:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.557099:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.557105:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.557120:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760789.557361:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9ad80 - object 0x280000401:2 offset 80039936<>80044032 00000100:00100000:0.0:1789760789.557403:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760789.557421:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503890816:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760789.572533:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986848 gen 2 00000008:00010000:0.0:1789760789.572566:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:0.0:1789760789.572572:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.572579:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.572600:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:0.0:1789760789.572634:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503890816:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760789.572818:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19541, 19541] 00000080:00200000:3.0:1789760789.572831:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.572836:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.572847:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80044032 i_blocks 156360 00000020:00200000:3.0:1789760789.572850:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.572859:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.572884:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80044032, count: 4096 00000080:00200000:3.0:1789760789.572890:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80044032, bytes: 4096 00000080:00200000:3.0:1789760789.572898:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.572905:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760789.572907:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80044032, 80048128) 0 0 00000080:00200000:3.0:1789760789.572911:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.572925:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19542, 19542] 00020000:00200000:3.0:1789760789.572931:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80044032 chunk: [80044032, 80048128) 80048128, 4096 00020000:00200000:3.0:1789760789.572935:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.572950:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19542, 19542] 00000008:00010000:3.0:1789760789.572964:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.572968:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760789.572977:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80044032, 80048128) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.572988:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80044032=0x4c56000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.573029:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.573064:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80048128 00000020:00010000:3.0:1789760789.573071:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.573074:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.573078:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.573081:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.573084:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.573090:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.573096:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.573109:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760789.573196:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80044032<>80048128 00000100:00100000:3.0:1789760789.586383:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986849 gen 2 00000008:00010000:3.0:1789760789.586419:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:3.0:1789760789.586425:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.586433:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.586454:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:3.0:1789760789.586477:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503890944:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.586602:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19542, 19542] 00000080:00200000:0.0:1789760789.586623:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.586628:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.586639:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80048128 i_blocks 156368 00000020:00200000:0.0:1789760789.586643:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.586660:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.586685:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80048128, count: 4096 00000080:00200000:0.0:1789760789.586692:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80048128, bytes: 4096 00000080:00200000:0.0:1789760789.586699:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.586705:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.586707:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80048128, 80052224) 0 0 00000080:00200000:0.0:1789760789.586711:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.586724:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19543, 19543] 00020000:00200000:0.0:1789760789.586731:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80048128 chunk: [80048128, 80052224) 80052224, 4096 00020000:00200000:0.0:1789760789.586735:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.586750:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19543, 19543] 00000008:00010000:0.0:1789760789.586763:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.586767:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.586775:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80048128, 80052224) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.586785:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80048128=0x4c57000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.587634:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.587639:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80052224 00000020:00010000:0.0:1789760789.587706:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.587710:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.587712:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.587717:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.587719:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.587728:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.587734:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.587795:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:3.0:1789760789.587877:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80048128<>80052224 00000100:00100000:3.0:1789760789.605102:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986850 gen 2 00000008:00010000:3.0:1789760789.605141:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:3.0:1789760789.605148:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.605158:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.605282:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:3.0:1789760789.605307:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503891072:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.605405:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19543, 19543] 00000080:00200000:0.0:1789760789.605418:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.605423:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.605435:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80052224 i_blocks 156376 00000020:00200000:0.0:1789760789.605438:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.605446:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.605471:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80052224, count: 4096 00000080:00200000:0.0:1789760789.605476:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80052224, bytes: 4096 00000080:00200000:0.0:1789760789.605485:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.605707:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.605709:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80052224, 80056320) 0 0 00000080:00200000:0.0:1789760789.605712:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.605726:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19544, 19544] 00020000:00200000:0.0:1789760789.605732:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80052224 chunk: [80052224, 80056320) 80056320, 4096 00020000:00200000:0.0:1789760789.605736:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.605749:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19544, 19544] 00000008:00010000:0.0:1789760789.605761:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.605766:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.605774:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80052224, 80056320) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.605785:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80052224=0x4c58000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.605825:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.605828:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80056320 00000020:00010000:0.0:1789760789.605835:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.605838:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.605840:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.605843:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.605845:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.605853:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.605859:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.605874:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760789.605949:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9ad80 - object 0x280000401:2 offset 80052224<>80056320 00000100:00100000:1.0:1789760789.606018:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503891200:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.622225:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986851 gen 2 00000008:00010000:1.0:1789760789.622265:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760789.622272:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.622283:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.622310:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760789.622345:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503891200:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.622642:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19544, 19544] 00000080:00200000:0.0:1789760789.622664:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.622669:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.622681:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80056320 i_blocks 156384 00000020:00200000:0.0:1789760789.622684:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.622692:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.622717:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80056320, count: 4096 00000080:00200000:0.0:1789760789.622723:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80056320, bytes: 4096 00000080:00200000:0.0:1789760789.622734:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.622741:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.622743:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80056320, 80060416) 0 0 00000080:00200000:0.0:1789760789.622746:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.622758:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19545, 19545] 00020000:00200000:0.0:1789760789.622765:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80056320 chunk: [80056320, 80060416) 80060416, 4096 00020000:00200000:0.0:1789760789.622769:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.622783:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19545, 19545] 00000008:00010000:0.0:1789760789.622797:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.622802:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.622810:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80056320, 80060416) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.622822:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80056320=0x4c59000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.622867:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.622870:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80060416 00000020:00010000:0.0:1789760789.622877:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.622880:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.622883:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.622885:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.622888:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.622896:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.622902:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.622915:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760789.622990:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98380 - object 0x280000401:2 offset 80056320<>80060416 00000100:00100000:1.0:1789760789.623421:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503891328:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.652987:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986852 gen 2 00000008:00010000:1.0:1789760789.653025:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760789.653031:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.653084:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.653094:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760789.653119:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503891328:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.654403:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19545, 19545] 00000080:00200000:0.0:1789760789.654432:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.654444:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.655463:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80060416 i_blocks 156392 00000020:00200000:0.0:1789760789.655471:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.655484:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.655512:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80060416, count: 4096 00000080:00200000:0.0:1789760789.655519:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80060416, bytes: 4096 00000080:00200000:0.0:1789760789.655530:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.655537:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.655539:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80060416, 80064512) 0 0 00000080:00200000:0.0:1789760789.655543:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.655556:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19546, 19546] 00020000:00200000:0.0:1789760789.655564:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80060416 chunk: [80060416, 80064512) 80064512, 4096 00020000:00200000:0.0:1789760789.655568:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.655584:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19546, 19546] 00000008:00010000:0.0:1789760789.655599:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.655615:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.655626:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80060416, 80064512) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.655637:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80060416=0x4c5a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.655691:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.655694:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80064512 00000020:00010000:0.0:1789760789.655710:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.655713:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.655716:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.655720:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.655722:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.655731:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.655737:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.655757:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760789.655841:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99c00 - object 0x280000401:2 offset 80060416<>80064512 00000100:00100000:0.0:1789760789.655892:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760789.655910:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503891456:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760789.682944:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986853 gen 2 00000008:00010000:0.0:1789760789.682984:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:0.0:1789760789.682991:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.683003:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.683127:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:0.0:1789760789.683154:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503891456:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760789.683507:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19546, 19546] 00000080:00200000:1.0:1789760789.683518:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.683523:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.683535:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80064512 i_blocks 156400 00000020:00200000:1.0:1789760789.683539:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.683546:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.683570:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80064512, count: 4096 00000080:00200000:1.0:1789760789.683575:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80064512, bytes: 4096 00000080:00200000:1.0:1789760789.683585:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.683592:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760789.683594:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80064512, 80068608) 0 0 00000080:00200000:1.0:1789760789.683598:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.683620:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19547, 19547] 00020000:00200000:1.0:1789760789.683627:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80064512 chunk: [80064512, 80068608) 80068608, 4096 00020000:00200000:1.0:1789760789.684257:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.684276:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19547, 19547] 00000008:00010000:1.0:1789760789.684289:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.684294:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760789.684302:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80064512, 80068608) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.684313:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80064512=0x4c5b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.684361:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.684363:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80068608 00000020:00010000:1.0:1789760789.684370:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.684373:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.684375:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.684378:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.684381:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.684387:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.684392:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.684505:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760789.684584:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 80064512<>80068608 00000100:00100000:0.0:1789760789.684685:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503891584:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.698210:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986854 gen 2 00000008:00010000:1.0:1789760789.698254:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:1.0:1789760789.698261:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.698270:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.698287:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000100:00100000:1.0:1789760789.698309:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503891584:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.698313:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19547, 19547] 00000080:00200000:0.0:1789760789.698386:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.698393:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.698407:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80068608 i_blocks 156408 00000020:00200000:0.0:1789760789.698411:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.698419:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.698443:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80068608, count: 4096 00000080:00200000:0.0:1789760789.698448:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80068608, bytes: 4096 00000080:00200000:0.0:1789760789.698456:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.698463:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.698465:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80068608, 80072704) 0 0 00000080:00200000:0.0:1789760789.698469:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.698482:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19548, 19548] 00020000:00200000:0.0:1789760789.698489:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80068608 chunk: [80068608, 80072704) 80072704, 4096 00020000:00200000:0.0:1789760789.698492:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.698507:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19548, 19548] 00000008:00010000:0.0:1789760789.698522:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.698527:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.698535:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80068608, 80072704) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.698546:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80068608=0x4c5c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.699259:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.699264:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80072704 00000020:00010000:0.0:1789760789.699272:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.699276:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.699279:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.699282:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.699284:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.699292:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.699298:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.699316:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760789.699398:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99c00 - object 0x280000401:2 offset 80068608<>80072704 00000100:00100000:1.0:1789760789.699484:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760789.699501:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503891712:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.712260:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986855 gen 2 00000008:00010000:1.0:1789760789.712296:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760789.712304:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.712316:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.712326:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760789.712352:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503891712:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.712537:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19548, 19548] 00000080:00200000:0.0:1789760789.712551:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.712557:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.712570:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80072704 i_blocks 156416 00000020:00200000:0.0:1789760789.712574:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.712582:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.712623:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80072704, count: 4096 00000080:00200000:0.0:1789760789.712631:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80072704, bytes: 4096 00000080:00200000:0.0:1789760789.712639:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.712647:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.712650:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80072704, 80076800) 0 0 00000080:00200000:0.0:1789760789.712731:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.712747:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19549, 19549] 00020000:00200000:0.0:1789760789.712755:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80072704 chunk: [80072704, 80076800) 80076800, 4096 00020000:00200000:0.0:1789760789.712759:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.712777:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19549, 19549] 00000008:00010000:0.0:1789760789.712793:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.712798:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.712809:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80072704, 80076800) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.712821:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80072704=0x4c5d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.712871:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.712873:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80076800 00000020:00010000:0.0:1789760789.712880:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.712883:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.712886:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.712888:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.712891:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.712898:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.712903:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.712918:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760789.712996:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a300 - object 0x280000401:2 offset 80072704<>80076800 00000100:00100000:1.0:1789760789.713147:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.713164:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503891840:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.733096:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986856 gen 2 00000008:00010000:1.0:1789760789.733146:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760789.733153:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.733163:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.733173:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760789.733196:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503891840:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.733883:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19549, 19549] 00000080:00200000:0.0:1789760789.733898:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.733903:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.733917:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80076800 i_blocks 156424 00000020:00200000:0.0:1789760789.733921:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.733929:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.733952:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80076800, count: 4096 00000080:00200000:0.0:1789760789.733959:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80076800, bytes: 4096 00000080:00200000:0.0:1789760789.733967:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.733975:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.733977:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80076800, 80080896) 0 0 00000080:00200000:0.0:1789760789.733980:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.734007:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19550, 19550] 00020000:00200000:0.0:1789760789.734014:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80076800 chunk: [80076800, 80080896) 80080896, 4096 00020000:00200000:0.0:1789760789.734018:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.734066:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19550, 19550] 00000008:00010000:0.0:1789760789.734082:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.734087:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.734096:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80076800, 80080896) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.734108:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80076800=0x4c5e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.734164:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.734167:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80080896 00000020:00010000:0.0:1789760789.734173:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.734176:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.734179:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.734182:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.734185:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.734192:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.734197:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.734211:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760789.734286:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99500 - object 0x280000401:2 offset 80076800<>80080896 00000100:00100000:1.0:1789760789.734465:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760789.734480:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503891968:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.749278:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986857 gen 2 00000008:00010000:1.0:1789760789.749320:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760789.749326:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.749336:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.749357:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760789.749382:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503891968:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.749528:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19550, 19550] 00000080:00200000:0.0:1789760789.749541:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.749545:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.749557:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80080896 i_blocks 156432 00000020:00200000:0.0:1789760789.749561:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.749568:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.749596:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80080896, count: 4096 00000080:00200000:0.0:1789760789.749602:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80080896, bytes: 4096 00000080:00200000:0.0:1789760789.749620:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.749628:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.749630:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80080896, 80084992) 0 0 00000080:00200000:0.0:1789760789.749633:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.749659:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19551, 19551] 00020000:00200000:0.0:1789760789.749667:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80080896 chunk: [80080896, 80084992) 80084992, 4096 00020000:00200000:0.0:1789760789.749671:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.749687:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19551, 19551] 00000008:00010000:0.0:1789760789.749702:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.749707:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.749717:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80080896, 80084992) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.749728:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80080896=0x4c5f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.749773:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.749776:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80084992 00000020:00010000:0.0:1789760789.749783:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.749786:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.749789:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.749792:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.749795:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.749802:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.749807:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.749820:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760789.749902:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99500 - object 0x280000401:2 offset 80080896<>80084992 00000100:00100000:1.0:1789760789.750111:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.750138:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503892096:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.775174:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986858 gen 2 00000008:00010000:1.0:1789760789.775211:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760789.775217:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.775226:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.775247:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760789.775272:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503892096:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.775582:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19551, 19551] 00000080:00200000:0.0:1789760789.775595:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.775600:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.775622:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80084992 i_blocks 156440 00000020:00200000:0.0:1789760789.775625:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.775635:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.775680:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80084992, count: 4096 00000080:00200000:0.0:1789760789.775686:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80084992, bytes: 4096 00000080:00200000:0.0:1789760789.775694:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.775701:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.775703:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80084992, 80089088) 0 0 00000080:00200000:0.0:1789760789.775708:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.775720:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19552, 19552] 00020000:00200000:0.0:1789760789.775727:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80084992 chunk: [80084992, 80089088) 80089088, 4096 00020000:00200000:0.0:1789760789.775731:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.775744:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19552, 19552] 00000008:00010000:0.0:1789760789.775757:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.775762:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.775771:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80084992, 80089088) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.775781:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80084992=0x4c60000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.775825:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.775828:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80089088 00000020:00010000:0.0:1789760789.775835:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.775838:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.775841:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.775845:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.775847:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.775855:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.775860:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.775876:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760789.775956:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a300 - object 0x280000401:2 offset 80084992<>80089088 00000100:00100000:1.0:1789760789.776301:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760789.776317:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503892224:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.805882:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986859 gen 2 00000008:00010000:1.0:1789760789.805919:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760789.805925:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.805935:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.805955:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760789.805981:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503892224:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.806137:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19552, 19552] 00000080:00200000:0.0:1789760789.806149:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.806153:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.806175:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80089088 i_blocks 156448 00000020:00200000:0.0:1789760789.806180:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.806187:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.806214:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80089088, count: 4096 00000080:00200000:0.0:1789760789.806220:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80089088, bytes: 4096 00000080:00200000:0.0:1789760789.806230:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.806236:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.806238:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80089088, 80093184) 0 0 00000080:00200000:0.0:1789760789.806241:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.806253:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19553, 19553] 00020000:00200000:0.0:1789760789.806259:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80089088 chunk: [80089088, 80093184) 80093184, 4096 00020000:00200000:0.0:1789760789.806263:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.806277:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19553, 19553] 00000008:00010000:0.0:1789760789.806290:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.806295:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.806304:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80089088, 80093184) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.806315:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80089088=0x4c61000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.806370:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.806373:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80093184 00000020:00010000:0.0:1789760789.806380:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.806382:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.806385:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.806388:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.806390:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.806397:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.806402:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.806416:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760789.806498:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99c00 - object 0x280000401:2 offset 80089088<>80093184 00000100:00100000:1.0:1789760789.806829:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.806855:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503892352:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.818109:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986860 gen 2 00000008:00010000:1.0:1789760789.818148:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760789.818154:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.818176:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.818193:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760789.818219:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503892352:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.818550:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19553, 19553] 00000080:00200000:0.0:1789760789.818563:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.818568:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.818580:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80093184 i_blocks 156456 00000020:00200000:0.0:1789760789.818583:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.818591:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.818627:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80093184, count: 4096 00000080:00200000:0.0:1789760789.818634:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80093184, bytes: 4096 00000080:00200000:0.0:1789760789.818641:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.818649:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.818651:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80093184, 80097280) 0 0 00000080:00200000:0.0:1789760789.818725:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.818740:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19554, 19554] 00020000:00200000:0.0:1789760789.818747:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80093184 chunk: [80093184, 80097280) 80097280, 4096 00020000:00200000:0.0:1789760789.818750:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.818766:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19554, 19554] 00000008:00010000:0.0:1789760789.818779:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.818785:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.818793:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80093184, 80097280) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.818803:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80093184=0x4c62000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.818919:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.818922:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80097280 00000020:00010000:0.0:1789760789.818930:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.818933:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.818935:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.818938:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.818941:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.818948:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.818954:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.818968:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760789.819080:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98380 - object 0x280000401:2 offset 80093184<>80097280 00000100:00100000:1.0:1789760789.819157:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760789.819181:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503892480:192.168.203.158@tcp:4:dd.0 00010000:00010000:0.0:1789760789.822939:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760789.840371:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986861 gen 2 00000008:00010000:1.0:1789760789.840417:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760789.840424:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.840433:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.840454:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760789.840477:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503892480:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.840973:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19554, 19554] 00000080:00200000:0.0:1789760789.840986:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.840991:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.841004:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80097280 i_blocks 156464 00000020:00200000:0.0:1789760789.841007:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.841015:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.841079:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80097280, count: 4096 00000080:00200000:0.0:1789760789.841086:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80097280, bytes: 4096 00000080:00200000:0.0:1789760789.841094:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.841101:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.841103:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80097280, 80101376) 0 0 00000080:00200000:0.0:1789760789.841106:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.841119:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19555, 19555] 00020000:00200000:0.0:1789760789.841126:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80097280 chunk: [80097280, 80101376) 80101376, 4096 00020000:00200000:0.0:1789760789.841130:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.841145:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19555, 19555] 00000008:00010000:0.0:1789760789.841159:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.841164:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.841173:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80097280, 80101376) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.841183:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80097280=0x4c63000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.841227:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.841230:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80101376 00000020:00010000:0.0:1789760789.841237:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.841240:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.841243:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.841247:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.841249:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.841256:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.841261:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.841276:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760789.841349:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9ad80 - object 0x280000401:2 offset 80097280<>80101376 00000100:00100000:1.0:1789760789.841563:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.841578:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503892608:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.858468:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986862 gen 2 00000008:00010000:1.0:1789760789.858513:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760789.858520:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.858528:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.858594:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760789.858627:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503892608:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.859732:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19555, 19555] 00000080:00200000:0.0:1789760789.859746:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.859752:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.859764:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80101376 i_blocks 156472 00000020:00200000:0.0:1789760789.859769:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.859778:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.859801:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80101376, count: 4096 00000080:00200000:0.0:1789760789.859807:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80101376, bytes: 4096 00000080:00200000:0.0:1789760789.859816:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.859822:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.859825:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80101376, 80105472) 0 0 00000080:00200000:0.0:1789760789.859828:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.859840:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19556, 19556] 00020000:00200000:0.0:1789760789.859846:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80101376 chunk: [80101376, 80105472) 80105472, 4096 00020000:00200000:0.0:1789760789.859850:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.859917:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19556, 19556] 00000008:00010000:0.0:1789760789.859931:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.859935:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.859945:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80101376, 80105472) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.859954:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80101376=0x4c64000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.860001:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.860003:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80105472 00000020:00010000:0.0:1789760789.860010:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.860013:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.860015:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.860018:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.860021:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.860028:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.860107:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.860126:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760789.860202:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b100 - object 0x280000401:2 offset 80101376<>80105472 00000100:00100000:1.0:1789760789.860259:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503892736:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.872218:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986863 gen 2 00000008:00010000:1.0:1789760789.872253:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760789.872260:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.872269:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.872289:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760789.872313:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503892736:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760789.872457:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19556, 19556] 00000080:00200000:0.0:1789760789.872469:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.872475:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.872487:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80105472 i_blocks 156480 00000020:00200000:0.0:1789760789.872490:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.872498:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.872523:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80105472, count: 4096 00000080:00200000:0.0:1789760789.872528:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80105472, bytes: 4096 00000080:00200000:0.0:1789760789.872536:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.872542:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760789.872544:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80105472, 80109568) 0 0 00000080:00200000:0.0:1789760789.872548:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.872561:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19557, 19557] 00020000:00200000:0.0:1789760789.872567:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80105472 chunk: [80105472, 80109568) 80109568, 4096 00020000:00200000:0.0:1789760789.872571:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.872587:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19557, 19557] 00000008:00010000:0.0:1789760789.872601:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.872616:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760789.872624:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80105472, 80109568) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.872634:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80105472=0x4c65000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.872685:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.872688:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80109568 00000020:00010000:0.0:1789760789.872694:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.872962:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.872966:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.872970:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.872972:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.872981:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.872987:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.873002:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760789.873600:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9aa00 - object 0x280000401:2 offset 80105472<>80109568 00000100:00100000:1.0:1789760789.873664:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.873866:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503892864:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.883935:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986864 gen 2 00000008:00010000:1.0:1789760789.883969:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760789.883976:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.883985:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.884068:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760789.884093:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503892864:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760789.884211:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19557, 19557] 00000080:00200000:3.0:1789760789.884222:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.884227:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.884238:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80109568 i_blocks 156488 00000020:00200000:3.0:1789760789.884241:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.884249:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.884272:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80109568, count: 4096 00000080:00200000:3.0:1789760789.884278:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80109568, bytes: 4096 00000080:00200000:3.0:1789760789.884285:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.884292:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760789.884293:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80109568, 80113664) 0 0 00000080:00200000:3.0:1789760789.884297:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.884309:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19558, 19558] 00020000:00200000:3.0:1789760789.884315:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80109568 chunk: [80109568, 80113664) 80113664, 4096 00020000:00200000:3.0:1789760789.884319:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.885090:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19558, 19558] 00000008:00010000:3.0:1789760789.885107:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.885111:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760789.885120:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80109568, 80113664) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.885131:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80109568=0x4c66000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.885174:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.885176:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80113664 00000020:00010000:3.0:1789760789.885182:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.885185:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.885188:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.885191:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.885194:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.885201:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.885206:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.885223:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760789.885297:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80109568<>80113664 00000100:00100000:3.0:1789760789.895194:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986865 gen 2 00000080:00200000:1.0:1789760789.896016:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19558, 19558] 00000080:00200000:1.0:1789760789.896028:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.896051:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.896064:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80113664 i_blocks 156496 00000020:00200000:1.0:1789760789.896067:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.896075:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.896100:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80113664, count: 4096 00000080:00200000:1.0:1789760789.896106:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80113664, bytes: 4096 00000080:00200000:1.0:1789760789.896116:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.896123:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760789.896124:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80113664, 80117760) 0 0 00000080:00200000:1.0:1789760789.896128:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.896141:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19559, 19559] 00020000:00200000:1.0:1789760789.896148:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80113664 chunk: [80113664, 80117760) 80117760, 4096 00020000:00200000:1.0:1789760789.896152:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.896167:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19559, 19559] 00000008:00010000:1.0:1789760789.896180:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.896185:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760789.896194:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80113664, 80117760) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.896204:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80113664=0x4c67000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.896314:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.896317:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80117760 00000020:00010000:1.0:1789760789.896325:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.896393:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.896396:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.896400:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.896402:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.896410:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.896415:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.896429:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760789.896568:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 80113664<>80117760 00000100:00100000:0.0:1789760789.896763:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503893120:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760789.910181:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986866 gen 2 00000008:00010000:0.0:1789760789.910216:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760789.910222:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.910230:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.910258:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000100:00100000:0.0:1789760789.910282:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503893120:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760789.910487:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19559, 19559] 00000080:00200000:3.0:1789760789.910500:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.910505:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.910518:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80117760 i_blocks 156504 00000020:00200000:3.0:1789760789.910522:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.910529:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.910552:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80117760, count: 4096 00000080:00200000:3.0:1789760789.910558:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80117760, bytes: 4096 00000080:00200000:3.0:1789760789.910566:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.910573:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760789.910575:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80117760, 80121856) 0 0 00000080:00200000:3.0:1789760789.910578:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.910590:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19560, 19560] 00020000:00200000:3.0:1789760789.910598:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80117760 chunk: [80117760, 80121856) 80121856, 4096 00020000:00200000:3.0:1789760789.910611:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.910628:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19560, 19560] 00000008:00010000:3.0:1789760789.910770:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.910775:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760789.910786:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80117760, 80121856) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.910796:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80117760=0x4c68000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.910840:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.910843:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80121856 00000020:00010000:3.0:1789760789.910849:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.910853:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.910855:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.910859:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.910862:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.910868:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.910873:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.910887:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760789.910962:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80117760<>80121856 00000100:00100000:3.0:1789760789.920638:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986867 gen 2 00000008:00010000:3.0:1789760789.920686:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760789.920692:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.920701:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.920721:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760789.920743:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503893248:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760789.920843:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19560, 19560] 00000080:00200000:1.0:1789760789.920857:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.920863:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.920876:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80121856 i_blocks 156512 00000020:00200000:1.0:1789760789.920879:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.920887:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.920916:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80121856, count: 4096 00000080:00200000:1.0:1789760789.920923:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80121856, bytes: 4096 00000080:00200000:1.0:1789760789.920931:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.920938:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760789.920940:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80121856, 80125952) 0 0 00000080:00200000:1.0:1789760789.920944:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.920957:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19561, 19561] 00020000:00200000:1.0:1789760789.920964:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80121856 chunk: [80121856, 80125952) 80125952, 4096 00020000:00200000:1.0:1789760789.920967:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.920981:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19561, 19561] 00000008:00010000:1.0:1789760789.920995:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.921000:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760789.921010:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80121856, 80125952) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.921020:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80121856=0x4c69000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.921093:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.921096:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80125952 00000020:00010000:1.0:1789760789.921103:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.921106:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.921108:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.921112:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.921114:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.921119:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.921124:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.921138:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760789.921220:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 80121856<>80125952 00000100:00100000:0.0:1789760789.921373:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503893376:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760789.944555:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986868 gen 2 00000008:00010000:0.0:1789760789.944592:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760789.944598:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.944619:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.944636:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000080:00200000:1.0:1789760789.944724:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19561, 19561] 00000080:00200000:1.0:1789760789.944736:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.944741:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.944754:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80125952 i_blocks 156520 00000020:00200000:1.0:1789760789.944758:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.944766:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.944793:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80125952, count: 4096 00000080:00200000:1.0:1789760789.944801:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80125952, bytes: 4096 00000080:00200000:1.0:1789760789.944807:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.944815:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760789.944817:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80125952, 80130048) 0 0 00000080:00200000:1.0:1789760789.944822:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.944835:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19562, 19562] 00020000:00200000:1.0:1789760789.944842:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80125952 chunk: [80125952, 80130048) 80130048, 4096 00020000:00200000:1.0:1789760789.944846:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.944862:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19562, 19562] 00000008:00010000:1.0:1789760789.944876:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.944882:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760789.944891:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80125952, 80130048) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.944901:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80125952=0x4c6a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.944947:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.944950:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80130048 00000020:00010000:1.0:1789760789.944956:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.944960:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.944963:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.944966:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.944969:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.944976:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.944981:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.944996:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760789.945106:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 80125952<>80130048 00000100:00100000:1.0:1789760789.945160:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503893504:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760789.945308:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503893376:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760789.973313:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986869 gen 2 00000008:00010000:1.0:1789760789.973363:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:1.0:1789760789.973370:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.973380:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.973403:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:1.0:1789760789.973427:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503893504:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760789.973565:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19562, 19562] 00000080:00200000:3.0:1789760789.973578:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.973584:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.973595:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80130048 i_blocks 156528 00000020:00200000:3.0:1789760789.973598:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.973616:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.973653:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80130048, count: 4096 00000080:00200000:3.0:1789760789.973660:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80130048, bytes: 4096 00000080:00200000:3.0:1789760789.973669:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.973675:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760789.973677:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80130048, 80134144) 0 0 00000080:00200000:3.0:1789760789.973680:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.973692:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19563, 19563] 00020000:00200000:3.0:1789760789.973699:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80130048 chunk: [80130048, 80134144) 80134144, 4096 00020000:00200000:3.0:1789760789.973702:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.973717:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19563, 19563] 00000008:00010000:3.0:1789760789.973730:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.973735:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760789.973744:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80130048, 80134144) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.973755:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80130048=0x4c6b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.973798:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.973802:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80134144 00000020:00010000:3.0:1789760789.973808:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.973811:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.973814:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.973817:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.973820:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.973826:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.973832:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.973846:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760789.973926:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80130048<>80134144 00000100:00100000:3.0:1789760789.973965:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760789.973978:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503893632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760789.984952:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986870 gen 2 00000008:00010000:3.0:1789760789.984984:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760789.984990:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.984997:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.985021:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760789.985076:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503893632:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760789.985479:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19563, 19563] 00000080:00200000:1.0:1789760789.985502:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.985508:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.985520:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80134144 i_blocks 156536 00000020:00200000:1.0:1789760789.985524:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.985532:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.985555:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80134144, count: 4096 00000080:00200000:1.0:1789760789.985561:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80134144, bytes: 4096 00000080:00200000:1.0:1789760789.985568:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.985576:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760789.985579:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80134144, 80138240) 0 0 00000080:00200000:1.0:1789760789.985583:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.985595:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19564, 19564] 00020000:00200000:1.0:1789760789.985612:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80134144 chunk: [80134144, 80138240) 80138240, 4096 00020000:00200000:1.0:1789760789.985617:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.985631:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19564, 19564] 00000008:00010000:1.0:1789760789.985645:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.985765:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760789.985776:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80134144, 80138240) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.985788:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80134144=0x4c6c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.985835:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.985838:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80138240 00000020:00010000:1.0:1789760789.985844:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.985847:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.985850:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.985853:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.985856:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.985862:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.985868:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.985883:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760789.985957:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 80134144<>80138240 00000100:00100000:0.0:1789760789.986019:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503893760:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760789.994851:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986871 gen 2 00000008:00010000:0.0:1789760789.994887:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760789.994893:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.994903:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.994923:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760789.994946:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503893760:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760789.995092:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19564, 19564] 00000080:00200000:1.0:1789760789.995104:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.995109:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.995120:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80138240 i_blocks 156544 00000020:00200000:1.0:1789760789.995124:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.995131:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.995155:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80138240, count: 4096 00000080:00200000:1.0:1789760789.995161:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80138240, bytes: 4096 00000080:00200000:1.0:1789760789.995168:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.995175:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760789.995178:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80138240, 80142336) 0 0 00000080:00200000:1.0:1789760789.995182:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.995194:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19565, 19565] 00020000:00200000:1.0:1789760789.995201:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80138240 chunk: [80138240, 80142336) 80142336, 4096 00020000:00200000:1.0:1789760789.995204:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.995218:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19565, 19565] 00000008:00010000:1.0:1789760789.995231:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.995236:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760789.995244:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80138240, 80142336) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.995256:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80138240=0x4c6d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.995298:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.995301:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80142336 00000020:00010000:1.0:1789760789.995307:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.995310:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.995313:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.995316:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.995319:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.995464:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.995471:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.995485:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760789.995571:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 80138240<>80142336 00000100:00100000:0.0:1789760789.995981:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760789.995996:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503893888:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.004198:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986872 gen 2 00000008:00010000:0.0:1789760790.004237:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:0.0:1789760790.004243:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.004251:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.004271:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:0.0:1789760790.004295:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503893888:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760790.004516:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19565, 19565] 00000080:00200000:3.0:1789760790.004529:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760790.004534:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760790.004546:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80142336 i_blocks 156552 00000020:00200000:3.0:1789760790.004549:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760790.004558:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760790.004582:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80142336, count: 4096 00000080:00200000:3.0:1789760790.004588:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80142336, bytes: 4096 00000080:00200000:3.0:1789760790.004597:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760790.004615:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760790.004617:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80142336, 80146432) 0 0 00000080:00200000:3.0:1789760790.004621:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760790.004633:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19566, 19566] 00020000:00200000:3.0:1789760790.004640:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80142336 chunk: [80142336, 80146432) 80146432, 4096 00020000:00200000:3.0:1789760790.004644:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760790.004667:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19566, 19566] 00000008:00010000:3.0:1789760790.004679:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760790.004683:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760790.004691:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80142336, 80146432) append 0 append lockpos 0 00000080:00200000:3.0:1789760790.004704:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80142336=0x4c6e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760790.004748:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760790.004751:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80146432 00000020:00010000:3.0:1789760790.004757:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760790.004761:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.004763:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.004766:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760790.004769:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760790.004774:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760790.004780:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.004793:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760790.004870:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80142336<>80146432 00000100:00100000:3.0:1789760790.018990:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986873 gen 2 00000008:00010000:3.0:1789760790.019074:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760790.019081:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.019092:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.019111:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760790.019136:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503894016:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.019235:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19566, 19566] 00000080:00200000:1.0:1789760790.019248:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.019253:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.019265:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80146432 i_blocks 156560 00000020:00200000:1.0:1789760790.019269:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.019276:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.019303:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80146432, count: 4096 00000080:00200000:1.0:1789760790.019310:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80146432, bytes: 4096 00000080:00200000:1.0:1789760790.019318:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.019326:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.019339:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80146432, 80150528) 0 0 00000080:00200000:1.0:1789760790.019343:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.019355:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19567, 19567] 00020000:00200000:1.0:1789760790.019362:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80146432 chunk: [80146432, 80150528) 80150528, 4096 00020000:00200000:1.0:1789760790.019366:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.019381:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19567, 19567] 00000008:00010000:1.0:1789760790.019395:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.019400:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.019408:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80146432, 80150528) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.019418:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80146432=0x4c6f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.019463:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.019466:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80150528 00000020:00010000:1.0:1789760790.019472:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.019476:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.019478:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.019482:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.019485:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.019501:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.019508:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.019521:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58780 00000008:00100000:1.0:1789760790.019595:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 80146432<>80150528 00000100:00100000:0.0:1789760790.019757:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894144:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.027851:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986874 gen 2 00000008:00010000:0.0:1789760790.027888:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58780 00000020:00200000:0.0:1789760790.027894:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.027903:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.027922:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58780 00000100:00100000:0.0:1789760790.027947:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894144:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.028460:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19567, 19567] 00000080:00200000:1.0:1789760790.028474:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.028479:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.028499:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80150528 i_blocks 156568 00000020:00200000:1.0:1789760790.028503:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.028512:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.028536:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80150528, count: 4096 00000080:00200000:1.0:1789760790.028542:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80150528, bytes: 4096 00000080:00200000:1.0:1789760790.028551:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.028558:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.028561:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80150528, 80154624) 0 0 00000080:00200000:1.0:1789760790.028565:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.028577:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19568, 19568] 00020000:00200000:1.0:1789760790.028584:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80150528 chunk: [80150528, 80154624) 80154624, 4096 00020000:00200000:1.0:1789760790.028587:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.028612:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19568, 19568] 00000008:00010000:1.0:1789760790.028626:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.028632:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.028650:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80150528, 80154624) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.028662:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80150528=0x4c70000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.028708:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.028712:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80154624 00000020:00010000:1.0:1789760790.028719:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.028722:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.028724:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.028728:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.028730:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.028737:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.028742:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.028757:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760790.028833:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 80150528<>80154624 00000100:00100000:0.0:1789760790.028991:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760790.029008:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894272:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.043929:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986875 gen 2 00000008:00010000:0.0:1789760790.043974:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:0.0:1789760790.043981:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.043990:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.044015:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000100:00100000:0.0:1789760790.044076:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894272:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.044170:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19568, 19568] 00000080:00200000:1.0:1789760790.044184:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.044189:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.044201:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80154624 i_blocks 156576 00000020:00200000:1.0:1789760790.044205:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.044212:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.044238:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80154624, count: 4096 00000080:00200000:1.0:1789760790.044244:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80154624, bytes: 4096 00000080:00200000:1.0:1789760790.044252:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.044260:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.044263:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80154624, 80158720) 0 0 00000080:00200000:1.0:1789760790.044267:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.044279:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19569, 19569] 00020000:00200000:1.0:1789760790.044285:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80154624 chunk: [80154624, 80158720) 80158720, 4096 00020000:00200000:1.0:1789760790.044289:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.044303:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19569, 19569] 00000008:00010000:1.0:1789760790.044316:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.044321:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.044329:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80154624, 80158720) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.044340:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80154624=0x4c71000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.044380:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.044382:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80158720 00000020:00010000:1.0:1789760790.044389:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.044392:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.044395:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.044398:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.044401:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.044407:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.044412:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.044425:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760790.044498:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 80154624<>80158720 00000100:00100000:0.0:1789760790.044933:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894400:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.058917:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986876 gen 2 00000008:00010000:0.0:1789760790.058956:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58000 00000020:00200000:0.0:1789760790.058962:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.058971:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.058991:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58000 00000100:00100000:0.0:1789760790.059024:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894400:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.059253:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19569, 19569] 00000080:00200000:1.0:1789760790.059266:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.059271:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.059284:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80158720 i_blocks 156584 00000020:00200000:1.0:1789760790.059288:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.059296:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.059323:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80158720, count: 4096 00000080:00200000:1.0:1789760790.059339:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80158720, bytes: 4096 00000080:00200000:1.0:1789760790.059347:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.059355:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.059357:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80158720, 80162816) 0 0 00000080:00200000:1.0:1789760790.059361:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.059373:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19570, 19570] 00020000:00200000:1.0:1789760790.059379:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80158720 chunk: [80158720, 80162816) 80162816, 4096 00020000:00200000:1.0:1789760790.059383:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.059397:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19570, 19570] 00000008:00010000:1.0:1789760790.059410:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.059415:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.059423:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80158720, 80162816) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.059433:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80158720=0x4c72000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.059473:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.059476:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80162816 00000020:00010000:1.0:1789760790.059482:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.059485:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.059488:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.059503:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.059506:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.059513:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.059519:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.059533:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58cc0 00000008:00100000:1.0:1789760790.059618:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 80158720<>80162816 00000100:00100000:0.0:1789760790.060266:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760790.060285:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894528:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.074308:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986877 gen 2 00000008:00010000:0.0:1789760790.074349:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58cc0 00000020:00200000:0.0:1789760790.074356:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.074366:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.074379:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58cc0 00000100:00100000:0.0:1789760790.074402:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894528:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.074511:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19570, 19570] 00000080:00200000:1.0:1789760790.074525:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.074530:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.074542:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80162816 i_blocks 156592 00000020:00200000:1.0:1789760790.074545:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.074553:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.074576:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80162816, count: 4096 00000080:00200000:1.0:1789760790.074582:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80162816, bytes: 4096 00000080:00200000:1.0:1789760790.074590:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.074597:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.074610:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80162816, 80166912) 0 0 00000080:00200000:1.0:1789760790.074614:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.074627:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19571, 19571] 00020000:00200000:1.0:1789760790.074635:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80162816 chunk: [80162816, 80166912) 80166912, 4096 00020000:00200000:1.0:1789760790.074646:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.074662:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19571, 19571] 00000008:00010000:1.0:1789760790.074675:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.074680:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.074687:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80162816, 80166912) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.074699:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80162816=0x4c73000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.074740:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.074744:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80166912 00000020:00010000:1.0:1789760790.074750:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.074753:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.074756:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.074759:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.074762:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.074768:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.074773:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.074786:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760790.074861:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 80162816<>80166912 00000100:00100000:0.0:1789760790.075173:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894656:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.082383:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986878 gen 2 00000008:00010000:0.0:1789760790.082416:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:0.0:1789760790.082423:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.082432:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.082451:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:0.0:1789760790.082471:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894656:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.082740:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19571, 19571] 00000080:00200000:1.0:1789760790.082753:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.082758:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.082770:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80166912 i_blocks 156600 00000020:00200000:1.0:1789760790.082774:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.082782:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.082804:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80166912, count: 4096 00000080:00200000:1.0:1789760790.082810:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80166912, bytes: 4096 00000080:00200000:1.0:1789760790.082818:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.082825:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.082828:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80166912, 80171008) 0 0 00000080:00200000:1.0:1789760790.082831:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.082843:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19572, 19572] 00020000:00200000:1.0:1789760790.082849:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80166912 chunk: [80166912, 80171008) 80171008, 4096 00020000:00200000:1.0:1789760790.082853:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.082866:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19572, 19572] 00000008:00010000:1.0:1789760790.082879:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.082884:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.082892:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80166912, 80171008) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.082902:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80166912=0x4c74000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.082944:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.082947:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80171008 00000020:00010000:1.0:1789760790.082953:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.082956:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.082959:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.082973:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.082977:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.082984:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.082989:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.083001:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760790.083102:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 80166912<>80171008 00000100:00100000:0.0:1789760790.083237:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503894784:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.095738:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986879 gen 2 00000008:00010000:0.0:1789760790.095779:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d586c0 00000020:00200000:0.0:1789760790.095784:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.095798:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.095819:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d586c0 00000100:00100000:0.0:1789760790.095843:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503894784:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.095934:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19572, 19572] 00000080:00200000:1.0:1789760790.095947:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.095952:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.095975:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80171008 i_blocks 156608 00000020:00200000:1.0:1789760790.095979:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.095988:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.096012:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80171008, count: 4096 00000080:00200000:1.0:1789760790.096017:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80171008, bytes: 4096 00000080:00200000:1.0:1789760790.096025:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.096064:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.096068:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80171008, 80175104) 0 0 00000080:00200000:1.0:1789760790.096071:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.096083:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19573, 19573] 00020000:00200000:1.0:1789760790.096090:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80171008 chunk: [80171008, 80175104) 80175104, 4096 00020000:00200000:1.0:1789760790.096094:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.096110:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19573, 19573] 00000008:00010000:1.0:1789760790.096125:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.096130:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.096138:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80171008, 80175104) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.096149:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80171008=0x4c75000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.096191:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.096194:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80175104 00000020:00010000:1.0:1789760790.096200:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.096203:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.096206:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.096209:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.096212:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.096218:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.096223:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.096236:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760790.096313:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 80171008<>80175104 00000100:00100000:0.0:1789760790.096864:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894912:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.112476:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986880 gen 2 00000008:00010000:0.0:1789760790.112523:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58a80 00000020:00200000:0.0:1789760790.112530:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.112538:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.112556:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58a80 00000100:00100000:0.0:1789760790.112580:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503894912:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.113758:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19573, 19573] 00000080:00200000:1.0:1789760790.113773:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.113779:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.113791:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80175104 i_blocks 156616 00000020:00200000:1.0:1789760790.113795:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.113805:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.113829:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80175104, count: 4096 00000080:00200000:1.0:1789760790.113835:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80175104, bytes: 4096 00000080:00200000:1.0:1789760790.113844:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.113851:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.113854:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80175104, 80179200) 0 0 00000080:00200000:1.0:1789760790.113858:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.113870:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19574, 19574] 00020000:00200000:1.0:1789760790.113877:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80175104 chunk: [80175104, 80179200) 80179200, 4096 00020000:00200000:1.0:1789760790.113881:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.113896:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19574, 19574] 00000008:00010000:1.0:1789760790.113910:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.113914:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.113923:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80175104, 80179200) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.113933:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80175104=0x4c76000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.114011:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.114014:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80179200 00000020:00010000:1.0:1789760790.114021:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.114030:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.114068:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.114072:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.114075:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.114082:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.114087:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.114103:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760790.114176:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 80175104<>80179200 00000100:00100000:0.0:1789760790.114819:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503895040:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.141786:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986881 gen 2 00000008:00010000:0.0:1789760790.141824:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58540 00000020:00200000:0.0:1789760790.141830:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.141839:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.141858:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58540 00000100:00100000:0.0:1789760790.141887:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503895040:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.142191:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19574, 19574] 00000080:00200000:1.0:1789760790.142204:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.142208:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.142220:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80179200 i_blocks 156624 00000020:00200000:1.0:1789760790.142224:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.142233:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.142257:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80179200, count: 4096 00000080:00200000:1.0:1789760790.142264:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80179200, bytes: 4096 00000080:00200000:1.0:1789760790.142271:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.142278:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.142280:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80179200, 80183296) 0 0 00000080:00200000:1.0:1789760790.142284:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.142295:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19575, 19575] 00020000:00200000:1.0:1789760790.142301:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80179200 chunk: [80179200, 80183296) 80183296, 4096 00020000:00200000:1.0:1789760790.142305:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.142318:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19575, 19575] 00000008:00010000:1.0:1789760790.142343:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.142347:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.142356:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80179200, 80183296) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.142367:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80179200=0x4c77000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.142410:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.142413:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80183296 00000020:00010000:1.0:1789760790.142419:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.142423:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.142426:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.142429:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.142432:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.142438:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.142443:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.142456:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760790.142529:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 80179200<>80183296 00000100:00100000:0.0:1789760790.142681:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503895168:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.157454:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986882 gen 2 00000008:00010000:0.0:1789760790.157499:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d583c0 00000020:00200000:0.0:1789760790.157505:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.157514:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.157531:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d583c0 00000100:00100000:0.0:1789760790.157557:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503895168:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.157827:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19575, 19575] 00000080:00200000:1.0:1789760790.157840:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.157845:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.157858:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80183296 i_blocks 156632 00000020:00200000:1.0:1789760790.157862:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.157870:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.157897:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80183296, count: 4096 00000080:00200000:1.0:1789760790.157904:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80183296, bytes: 4096 00000080:00200000:1.0:1789760790.157913:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.157930:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.157933:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80183296, 80187392) 0 0 00000080:00200000:1.0:1789760790.157936:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.157949:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19576, 19576] 00020000:00200000:1.0:1789760790.157955:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80183296 chunk: [80183296, 80187392) 80187392, 4096 00020000:00200000:1.0:1789760790.157959:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.157974:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19576, 19576] 00000008:00010000:1.0:1789760790.157987:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.157992:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.158000:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80183296, 80187392) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.158010:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80183296=0x4c78000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.158076:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.158080:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80187392 00000020:00010000:1.0:1789760790.158086:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.158089:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.158092:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.158094:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.158097:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.158103:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.158108:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.158122:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760790.158194:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 80183296<>80187392 00000100:00100000:0.0:1789760790.158479:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503895296:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.171633:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986883 gen 2 00000008:00010000:0.0:1789760790.171739:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:0.0:1789760790.171746:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.171756:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.171775:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:0.0:1789760790.171798:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503895296:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.171891:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19576, 19576] 00000080:00200000:1.0:1789760790.171902:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.171907:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.171992:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80187392 i_blocks 156640 00000020:00200000:1.0:1789760790.171997:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.172005:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.172030:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80187392, count: 4096 00000080:00200000:1.0:1789760790.172059:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80187392, bytes: 4096 00000080:00200000:1.0:1789760790.172066:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.172072:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.172075:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80187392, 80191488) 0 0 00000080:00200000:1.0:1789760790.172078:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.172091:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19577, 19577] 00020000:00200000:1.0:1789760790.172097:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80187392 chunk: [80187392, 80191488) 80191488, 4096 00020000:00200000:1.0:1789760790.172101:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.172116:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19577, 19577] 00000008:00010000:1.0:1789760790.172130:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.172135:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.172144:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80187392, 80191488) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.172154:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80187392=0x4c79000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.172198:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.172201:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80191488 00000020:00010000:1.0:1789760790.172207:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.172211:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.172214:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.172218:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.172221:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.172227:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.172233:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.172247:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760790.172554:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 80187392<>80191488 00000100:00100000:0.0:1789760790.172724:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503895424:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.186065:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986884 gen 2 00000008:00010000:0.0:1789760790.186099:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:0.0:1789760790.186105:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.186115:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.186134:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:0.0:1789760790.186159:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503895424:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.186616:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19577, 19577] 00000080:00200000:1.0:1789760790.186630:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.186635:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.186718:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80191488 i_blocks 156648 00000020:00200000:1.0:1789760790.186723:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.186732:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.186759:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80191488, count: 4096 00000080:00200000:1.0:1789760790.186765:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80191488, bytes: 4096 00000080:00200000:1.0:1789760790.186773:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.186780:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.186783:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80191488, 80195584) 0 0 00000080:00200000:1.0:1789760790.186786:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.186799:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19578, 19578] 00020000:00200000:1.0:1789760790.186806:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80191488 chunk: [80191488, 80195584) 80195584, 4096 00020000:00200000:1.0:1789760790.186810:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.186825:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19578, 19578] 00000008:00010000:1.0:1789760790.186839:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.186844:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.186853:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80191488, 80195584) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.186863:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80191488=0x4c7a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.186910:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.186913:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80195584 00000020:00010000:1.0:1789760790.187467:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.187473:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.187476:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.187480:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.187483:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.187490:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.187496:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.187515:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760790.187594:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 80191488<>80195584 00000100:00100000:0.0:1789760790.187796:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503895552:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.200195:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986885 gen 2 00000008:00010000:0.0:1789760790.200233:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d583c0 00000020:00200000:0.0:1789760790.200239:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.200248:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.200654:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d583c0 00000100:00100000:0.0:1789760790.200679:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503895552:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.201097:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19578, 19578] 00000080:00200000:1.0:1789760790.201109:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.201114:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.201125:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80195584 i_blocks 156656 00000020:00200000:1.0:1789760790.201128:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.201137:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.201160:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80195584, count: 4096 00000080:00200000:1.0:1789760790.201167:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80195584, bytes: 4096 00000080:00200000:1.0:1789760790.201175:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.201182:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.201185:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80195584, 80199680) 0 0 00000080:00200000:1.0:1789760790.201188:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.201200:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19579, 19579] 00020000:00200000:1.0:1789760790.201207:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80195584 chunk: [80195584, 80199680) 80199680, 4096 00020000:00200000:1.0:1789760790.201210:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.201224:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19579, 19579] 00000008:00010000:1.0:1789760790.201237:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.201242:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.201250:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80195584, 80199680) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.201438:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80195584=0x4c7b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.201483:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.201486:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80199680 00000020:00010000:1.0:1789760790.201503:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.201507:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.201510:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.201513:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.201516:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.201522:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.201528:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.201542:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760790.201625:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 80195584<>80199680 00000100:00100000:0.0:1789760790.201741:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503895680:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.217007:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986886 gen 2 00000008:00010000:0.0:1789760790.217075:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58540 00000020:00200000:0.0:1789760790.217081:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.217090:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.217100:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58540 00000100:00100000:0.0:1789760790.217122:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503895680:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.217474:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19579, 19579] 00000080:00200000:1.0:1789760790.217487:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.217502:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.217513:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80199680 i_blocks 156664 00000020:00200000:1.0:1789760790.217517:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.217525:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.217549:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80199680, count: 4096 00000080:00200000:1.0:1789760790.217555:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80199680, bytes: 4096 00000080:00200000:1.0:1789760790.217563:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.217570:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.217573:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80199680, 80203776) 0 0 00000080:00200000:1.0:1789760790.217578:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.217590:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19580, 19580] 00020000:00200000:1.0:1789760790.217597:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80199680 chunk: [80199680, 80203776) 80203776, 4096 00020000:00200000:1.0:1789760790.217611:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.217630:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19580, 19580] 00000008:00010000:1.0:1789760790.217644:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.217648:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.217657:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80199680, 80203776) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.217668:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80199680=0x4c7c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.217712:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.217714:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80203776 00000020:00010000:1.0:1789760790.217721:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.217724:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.217726:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.217729:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.217732:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.217738:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.217743:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.217757:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760790.217837:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 80199680<>80203776 00000100:00100000:0.0:1789760790.218103:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503895808:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.231462:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986887 gen 2 00000008:00010000:0.0:1789760790.231556:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58a80 00000020:00200000:0.0:1789760790.231563:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.231572:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.231592:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58a80 00000100:00100000:0.0:1789760790.231626:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503895808:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.231721:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19580, 19580] 00000080:00200000:1.0:1789760790.231733:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.231738:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.231750:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80203776 i_blocks 156672 00000020:00200000:1.0:1789760790.231753:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.231761:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.231785:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80203776, count: 4096 00000080:00200000:1.0:1789760790.231790:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80203776, bytes: 4096 00000080:00200000:1.0:1789760790.231798:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.231805:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.231807:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80203776, 80207872) 0 0 00000080:00200000:1.0:1789760790.231810:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.231822:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19581, 19581] 00020000:00200000:1.0:1789760790.231829:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80203776 chunk: [80203776, 80207872) 80207872, 4096 00020000:00200000:1.0:1789760790.231832:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.231848:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19581, 19581] 00000008:00010000:1.0:1789760790.231860:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.231865:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.231875:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80203776, 80207872) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.231885:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80203776=0x4c7d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.231926:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.231929:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80207872 00000020:00010000:1.0:1789760790.231935:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.231938:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.231941:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.231944:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.231947:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.231953:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.231958:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.231971:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760790.232141:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 80203776<>80207872 00000100:00100000:0.0:1789760790.232186:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503895936:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.245104:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986888 gen 2 00000008:00010000:0.0:1789760790.245142:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d586c0 00000020:00200000:0.0:1789760790.245149:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.245158:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.245181:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d586c0 00000100:00100000:0.0:1789760790.245207:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503895936:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760790.245313:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19581, 19581] 00000080:00200000:3.0:1789760790.245330:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760790.245336:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760790.245349:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80207872 i_blocks 156680 00000020:00200000:3.0:1789760790.245354:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760790.245362:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760790.245386:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80207872, count: 4096 00000080:00200000:3.0:1789760790.245393:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80207872, bytes: 4096 00000080:00200000:3.0:1789760790.245400:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760790.245407:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760790.245409:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80207872, 80211968) 0 0 00000080:00200000:3.0:1789760790.245413:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760790.245425:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19582, 19582] 00020000:00200000:3.0:1789760790.245432:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80207872 chunk: [80207872, 80211968) 80211968, 4096 00020000:00200000:3.0:1789760790.245435:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760790.245450:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19582, 19582] 00000008:00010000:3.0:1789760790.245462:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760790.245466:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760790.245475:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80207872, 80211968) append 0 append lockpos 0 00000080:00200000:3.0:1789760790.245485:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80207872=0x4c7e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760790.245538:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760790.245541:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80211968 00000020:00010000:3.0:1789760790.245548:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760790.245551:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.245554:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.245558:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760790.245561:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760790.245566:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760790.245572:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.245585:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760790.245670:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80207872<>80211968 00000100:00100000:3.0:1789760790.245709:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760790.245722:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503896064:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.254930:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986889 gen 2 00000008:00010000:3.0:1789760790.254964:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760790.254970:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.254978:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.255000:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760790.255023:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503896064:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.255999:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19582, 19582] 00000080:00200000:0.0:1789760790.256013:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.256018:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.256030:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80211968 i_blocks 156688 00000020:00200000:0.0:1789760790.256053:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.256063:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.256087:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80211968, count: 4096 00000080:00200000:0.0:1789760790.256093:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80211968, bytes: 4096 00000080:00200000:0.0:1789760790.256102:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.256110:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.256111:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80211968, 80216064) 0 0 00000080:00200000:0.0:1789760790.256115:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.256127:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19583, 19583] 00020000:00200000:0.0:1789760790.256133:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80211968 chunk: [80211968, 80216064) 80216064, 4096 00020000:00200000:0.0:1789760790.256137:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.256152:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19583, 19583] 00000008:00010000:0.0:1789760790.256166:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.256170:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.256179:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80211968, 80216064) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.256190:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80211968=0x4c7f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.256235:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.256238:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80216064 00000020:00010000:0.0:1789760790.256244:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.256248:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.256250:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.256427:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.256432:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.256441:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.256447:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.256462:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760790.256711:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b800 - object 0x280000401:2 offset 80211968<>80216064 00000100:00100000:1.0:1789760790.256776:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503896192:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.265331:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986890 gen 2 00000008:00010000:1.0:1789760790.265365:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760790.265372:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.265381:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.265452:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760790.265479:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503896192:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.266714:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503896320:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.277905:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986891 gen 2 00000008:00010000:3.0:1789760790.277941:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760790.277948:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.277956:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.277976:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760790.277998:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503896320:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.298867:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986892 gen 2 00000008:00010000:3.0:1789760790.298916:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760790.298922:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.298931:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.298951:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760790.298977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503896448:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.300236:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760790.300253:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503896576:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.318814:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986893 gen 2 00000008:00010000:3.0:1789760790.318848:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760790.318855:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.318862:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.318948:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760790.318971:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503896576:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.321027:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503896704:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.337725:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986894 gen 2 00000008:00010000:3.0:1789760790.337761:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760790.337767:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.337777:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.337802:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760790.337826:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503896704:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.339541:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503896832:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.355813:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986895 gen 2 00000008:00010000:3.0:1789760790.355848:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760790.355854:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.355863:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.355888:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760790.355912:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503896832:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.356493:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503896960:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.377400:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986896 gen 2 00000008:00010000:3.0:1789760790.377435:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760790.377443:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.377451:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.377472:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760790.377497:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503896960:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.399061:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986897 gen 2 00000008:00010000:3.0:1789760790.399099:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760790.399106:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.399117:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.399139:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760790.399166:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503897088:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.399789:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760790.399804:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503897216:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.413010:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986898 gen 2 00000008:00010000:3.0:1789760790.413140:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760790.413147:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.413155:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.413172:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760790.413195:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503897216:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.413827:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760790.413842:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503897344:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.426832:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986899 gen 2 00000008:00010000:3.0:1789760790.426885:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760790.426891:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.426900:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.426919:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760790.426943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503897344:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.427570:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760790.427584:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503897472:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.439217:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986900 gen 2 00000008:00010000:3.0:1789760790.439251:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760790.439272:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.439281:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.439303:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760790.439327:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503897472:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.439905:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760790.439919:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503897600:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.448982:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986901 gen 2 00000008:00010000:3.0:1789760790.449018:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760790.449024:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.449066:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.449131:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760790.449157:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503897600:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.452827:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503897728:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.464393:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986902 gen 2 00000008:00010000:3.0:1789760790.464432:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760790.464439:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.464447:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.464467:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760790.464502:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503897728:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.465477:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503897856:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.478089:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986903 gen 2 00000008:00010000:3.0:1789760790.478126:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760790.478133:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.478141:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.478166:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760790.478189:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503897856:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.478287:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19596, 19596] 00000080:00200000:0.0:1789760790.478300:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.478305:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.478317:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80269312 i_blocks 156800 00000020:00200000:0.0:1789760790.478334:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.478342:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.478368:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80269312, count: 4096 00000080:00200000:0.0:1789760790.478374:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80269312, bytes: 4096 00000080:00200000:0.0:1789760790.478382:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.478390:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.478392:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80269312, 80273408) 0 0 00000080:00200000:0.0:1789760790.478395:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.478408:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19597, 19597] 00020000:00200000:0.0:1789760790.478416:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80269312 chunk: [80269312, 80273408) 80273408, 4096 00020000:00200000:0.0:1789760790.478419:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.478434:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19597, 19597] 00000008:00010000:0.0:1789760790.478447:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.478452:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.478461:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80269312, 80273408) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.478470:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80269312=0x4c8d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.478519:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.478522:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80273408 00000020:00010000:0.0:1789760790.478527:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.478530:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.478533:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.478536:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.478539:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.478546:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.478551:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.478564:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760790.478809:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b480 - object 0x280000401:2 offset 80269312<>80273408 00000100:00100000:1.0:1789760790.478890:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503897984:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.489907:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986904 gen 2 00000008:00010000:1.0:1789760790.489947:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760790.489954:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.489963:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.489985:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760790.490010:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503897984:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.490414:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19597, 19597] 00000080:00200000:0.0:1789760790.490426:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.490430:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.490442:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80273408 i_blocks 156808 00000020:00200000:0.0:1789760790.490446:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.490454:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.490476:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80273408, count: 4096 00000080:00200000:0.0:1789760790.490482:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80273408, bytes: 4096 00000080:00200000:0.0:1789760790.490492:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.490499:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.490501:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80273408, 80277504) 0 0 00000080:00200000:0.0:1789760790.490513:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.490527:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19598, 19598] 00020000:00200000:0.0:1789760790.490534:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80273408 chunk: [80273408, 80277504) 80277504, 4096 00020000:00200000:0.0:1789760790.490538:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.490554:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19598, 19598] 00000008:00010000:0.0:1789760790.490567:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.490572:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.490580:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80273408, 80277504) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.490591:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80273408=0x4c8e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.490740:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.490744:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80277504 00000020:00010000:0.0:1789760790.490751:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.490754:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.490757:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.490760:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.490763:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.490770:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.490776:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.490789:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760790.490865:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a680 - object 0x280000401:2 offset 80273408<>80277504 00000100:00100000:1.0:1789760790.492151:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503898112:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.500953:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986905 gen 2 00000008:00010000:1.0:1789760790.500989:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760790.500994:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.501003:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.501023:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760790.501074:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503898112:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.501254:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19598, 19598] 00000080:00200000:0.0:1789760790.501265:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.501271:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.501282:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80277504 i_blocks 156816 00000020:00200000:0.0:1789760790.501285:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.501292:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.501314:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80277504, count: 4096 00000080:00200000:0.0:1789760790.501330:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80277504, bytes: 4096 00000080:00200000:0.0:1789760790.501338:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.501345:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.501347:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80277504, 80281600) 0 0 00000080:00200000:0.0:1789760790.501351:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.501363:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19599, 19599] 00020000:00200000:0.0:1789760790.501370:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80277504 chunk: [80277504, 80281600) 80281600, 4096 00020000:00200000:0.0:1789760790.501373:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.501388:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19599, 19599] 00000008:00010000:0.0:1789760790.501402:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.501406:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.501416:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80277504, 80281600) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.501427:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80277504=0x4c8f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.501471:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.501475:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80281600 00000020:00010000:0.0:1789760790.501482:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.501485:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.501488:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.501490:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.501494:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.501501:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.501506:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.501519:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760790.501592:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98000 - object 0x280000401:2 offset 80277504<>80281600 00000100:00100000:1.0:1789760790.501940:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503898240:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.512745:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986906 gen 2 00000008:00010000:1.0:1789760790.512778:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760790.512784:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.512793:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.512846:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19599, 19599] 00000008:00010000:1.0:1789760790.512846:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000080:00200000:0.0:1789760790.512859:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.512864:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:1.0:1789760790.512870:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503898240:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.512875:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80281600 i_blocks 156824 00000020:00200000:0.0:1789760790.512878:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.512886:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.512910:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80281600, count: 4096 00000080:00200000:0.0:1789760790.512916:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80281600, bytes: 4096 00000080:00200000:0.0:1789760790.513193:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.513200:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.513202:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80281600, 80285696) 0 0 00000080:00200000:0.0:1789760790.513205:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.513218:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19600, 19600] 00020000:00200000:0.0:1789760790.513225:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80281600 chunk: [80281600, 80285696) 80285696, 4096 00020000:00200000:0.0:1789760790.513228:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.513242:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19600, 19600] 00000008:00010000:0.0:1789760790.513255:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.513259:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.513267:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80281600, 80285696) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.513279:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80281600=0x4c90000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.513322:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.513325:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80285696 00000020:00010000:0.0:1789760790.513332:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.513335:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.513338:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.513341:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.513344:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.513351:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.513357:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.513370:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760790.513446:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98a80 - object 0x280000401:2 offset 80281600<>80285696 00000100:00100000:1.0:1789760790.513486:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760790.513500:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503898368:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.525610:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986907 gen 2 00000008:00010000:1.0:1789760790.525646:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760790.525652:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.525662:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.525684:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760790.525709:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503898368:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.526012:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19600, 19600] 00000080:00200000:0.0:1789760790.526025:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.526029:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.526066:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80285696 i_blocks 156832 00000020:00200000:0.0:1789760790.526070:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.526078:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.526299:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80285696, count: 4096 00000080:00200000:0.0:1789760790.526307:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80285696, bytes: 4096 00000080:00200000:0.0:1789760790.526316:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.526323:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.526325:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80285696, 80289792) 0 0 00000080:00200000:0.0:1789760790.526329:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.526343:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19601, 19601] 00020000:00200000:0.0:1789760790.526350:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80285696 chunk: [80285696, 80289792) 80289792, 4096 00020000:00200000:0.0:1789760790.526354:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.526368:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19601, 19601] 00000008:00010000:0.0:1789760790.526382:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.526387:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.526396:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80285696, 80289792) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.526408:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80285696=0x4c91000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.526455:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.526458:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80289792 00000020:00010000:0.0:1789760790.526464:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.526468:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.526471:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.526474:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.526476:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.526483:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.526489:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.526505:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760790.526583:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 80285696<>80289792 00000100:00100000:0.0:1789760790.526628:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760790.526643:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503898496:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.544789:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986908 gen 2 00000008:00010000:0.0:1789760790.544826:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:0.0:1789760790.544833:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.544842:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.544869:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:0.0:1789760790.544897:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503898496:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760790.545080:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19601, 19601] 00000080:00200000:3.0:1789760790.545094:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760790.545099:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760790.545188:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80289792 i_blocks 156840 00000020:00200000:3.0:1789760790.545193:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760790.545202:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760790.545228:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80289792, count: 4096 00000080:00200000:3.0:1789760790.545235:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80289792, bytes: 4096 00000080:00200000:3.0:1789760790.545246:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760790.545254:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760790.545256:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80289792, 80293888) 0 0 00000080:00200000:3.0:1789760790.545260:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760790.545273:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19602, 19602] 00020000:00200000:3.0:1789760790.545280:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80289792 chunk: [80289792, 80293888) 80293888, 4096 00020000:00200000:3.0:1789760790.545284:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760790.545300:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19602, 19602] 00000008:00010000:3.0:1789760790.545315:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760790.545321:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760790.545329:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80289792, 80293888) append 0 append lockpos 0 00000080:00200000:3.0:1789760790.545340:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80289792=0x4c92000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760790.545386:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760790.545388:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80293888 00000020:00010000:3.0:1789760790.545395:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760790.545398:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.545402:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.545405:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760790.545407:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760790.545414:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760790.545420:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.545433:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760790.545511:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80289792<>80293888 00000100:00100000:3.0:1789760790.557846:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986909 gen 2 00000008:00010000:3.0:1789760790.557884:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760790.557892:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.557901:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.557932:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760790.557955:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503898624:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.558125:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19602, 19602] 00000080:00200000:0.0:1789760790.558137:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.558142:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.558154:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80293888 i_blocks 156848 00000020:00200000:0.0:1789760790.558157:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.558165:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.558190:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80293888, count: 4096 00000080:00200000:0.0:1789760790.558196:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80293888, bytes: 4096 00000080:00200000:0.0:1789760790.558203:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.558209:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.558211:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80293888, 80297984) 0 0 00000080:00200000:0.0:1789760790.558214:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.558225:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19603, 19603] 00020000:00200000:0.0:1789760790.558232:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80293888 chunk: [80293888, 80297984) 80297984, 4096 00020000:00200000:0.0:1789760790.558235:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.558247:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19603, 19603] 00000008:00010000:0.0:1789760790.558259:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.558263:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.558271:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80293888, 80297984) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.558280:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80293888=0x4c93000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.558318:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.558386:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80297984 00000020:00010000:0.0:1789760790.558393:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.558396:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.558399:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.558402:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.558404:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.558411:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.558416:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.558428:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760790.558766:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 80293888<>80297984 00000100:00100000:1.0:1789760790.559024:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503898752:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.568766:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986910 gen 2 00000008:00010000:1.0:1789760790.568808:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760790.568815:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.568824:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.568844:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760790.568870:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503898752:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.568971:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19603, 19603] 00000080:00200000:0.0:1789760790.568984:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.568989:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.569001:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80297984 i_blocks 156856 00000020:00200000:0.0:1789760790.569004:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.569012:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.569064:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80297984, count: 4096 00000080:00200000:0.0:1789760790.569070:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80297984, bytes: 4096 00000080:00200000:0.0:1789760790.569078:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.569084:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.569087:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80297984, 80302080) 0 0 00000080:00200000:0.0:1789760790.569090:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.569101:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19604, 19604] 00020000:00200000:0.0:1789760790.569109:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80297984 chunk: [80297984, 80302080) 80302080, 4096 00020000:00200000:0.0:1789760790.569112:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.569128:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19604, 19604] 00000008:00010000:0.0:1789760790.569142:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.569147:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.569156:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80297984, 80302080) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.569167:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80297984=0x4c94000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.569209:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.569213:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80302080 00000020:00010000:0.0:1789760790.569220:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.569223:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.569225:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.569228:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.569230:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.569237:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.569242:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.569256:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760790.569341:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99880 - object 0x280000401:2 offset 80297984<>80302080 00000100:00100000:1.0:1789760790.569773:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760790.569789:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503898880:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.578656:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986911 gen 2 00000008:00010000:1.0:1789760790.578694:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760790.578701:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.578709:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.578730:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760790.578755:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503898880:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.578950:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19604, 19604] 00000080:00200000:0.0:1789760790.578961:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.578967:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.578979:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80302080 i_blocks 156864 00000020:00200000:0.0:1789760790.578983:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.578991:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.579018:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80302080, count: 4096 00000080:00200000:0.0:1789760790.579024:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80302080, bytes: 4096 00000080:00200000:0.0:1789760790.579031:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.579065:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.579067:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80302080, 80306176) 0 0 00000080:00200000:0.0:1789760790.579071:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.579084:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19605, 19605] 00020000:00200000:0.0:1789760790.579090:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80302080 chunk: [80302080, 80306176) 80306176, 4096 00020000:00200000:0.0:1789760790.579094:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.579108:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19605, 19605] 00000008:00010000:0.0:1789760790.579122:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.579127:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.579136:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80302080, 80306176) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.579147:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80302080=0x4c95000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.579190:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.579193:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80306176 00000020:00010000:0.0:1789760790.579200:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.579202:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.579205:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.579208:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.579211:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.579218:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.579223:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.579237:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760790.579316:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99f80 - object 0x280000401:2 offset 80302080<>80306176 00000100:00100000:1.0:1789760790.579528:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760790.579543:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503899008:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.594986:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986912 gen 2 00000008:00010000:1.0:1789760790.595070:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760790.595078:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.595087:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.595119:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760790.595146:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503899008:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.595688:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19605, 19605] 00000080:00200000:0.0:1789760790.595702:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.595708:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.595720:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80306176 i_blocks 156872 00000020:00200000:0.0:1789760790.595724:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.595732:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.595765:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80306176, count: 4096 00000080:00200000:0.0:1789760790.595771:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80306176, bytes: 4096 00000080:00200000:0.0:1789760790.595783:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.595790:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.595793:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80306176, 80310272) 0 0 00000080:00200000:0.0:1789760790.595797:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.595809:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19606, 19606] 00020000:00200000:0.0:1789760790.595816:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80306176 chunk: [80306176, 80310272) 80310272, 4096 00020000:00200000:0.0:1789760790.595819:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.595834:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19606, 19606] 00000008:00010000:0.0:1789760790.595848:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.595853:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.595861:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80306176, 80310272) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.595873:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80306176=0x4c96000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.595918:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.595922:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80310272 00000020:00010000:0.0:1789760790.595930:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.595933:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.595936:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.595939:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.595942:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.595949:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.595955:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.595969:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760790.596080:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9bb80 - object 0x280000401:2 offset 80306176<>80310272 00000100:00100000:1.0:1789760790.596504:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760790.596521:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503899136:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.610445:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986913 gen 2 00000008:00010000:1.0:1789760790.610486:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760790.610492:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.610502:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.610526:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760790.610551:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503899136:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.610670:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19606, 19606] 00000080:00200000:0.0:1789760790.610684:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.610690:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.610702:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80310272 i_blocks 156880 00000020:00200000:0.0:1789760790.610706:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.610714:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.610741:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80310272, count: 4096 00000080:00200000:0.0:1789760790.610759:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80310272, bytes: 4096 00000080:00200000:0.0:1789760790.610767:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.610775:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.610777:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80310272, 80314368) 0 0 00000080:00200000:0.0:1789760790.610781:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.610793:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19607, 19607] 00020000:00200000:0.0:1789760790.610801:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80310272 chunk: [80310272, 80314368) 80314368, 4096 00020000:00200000:0.0:1789760790.610805:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.610821:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19607, 19607] 00000008:00010000:0.0:1789760790.610835:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.610840:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.610848:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80310272, 80314368) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.610858:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80310272=0x4c97000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.610905:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.610909:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80314368 00000020:00010000:0.0:1789760790.610915:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.610918:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.610922:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.610925:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.610928:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.610936:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.610942:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.610957:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760790.611080:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98700 - object 0x280000401:2 offset 80310272<>80314368 00000100:00100000:1.0:1789760790.611158:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760790.611174:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503899264:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.632402:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986914 gen 2 00000008:00010000:1.0:1789760790.632439:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760790.632446:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.632456:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.632538:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760790.632565:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503899264:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.632725:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19607, 19607] 00000080:00200000:0.0:1789760790.632737:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.632742:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.632836:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80314368 i_blocks 156888 00000020:00200000:1.0:1789760790.632840:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.632848:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.632872:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80314368, count: 4096 00000080:00200000:1.0:1789760790.632878:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80314368, bytes: 4096 00000080:00200000:1.0:1789760790.632887:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.632893:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.632895:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80314368, 80318464) 0 0 00000080:00200000:1.0:1789760790.632898:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.632911:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19608, 19608] 00020000:00200000:1.0:1789760790.633234:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80314368 chunk: [80314368, 80318464) 80318464, 4096 00020000:00200000:1.0:1789760790.633239:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.633256:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19608, 19608] 00000008:00010000:1.0:1789760790.633269:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.633274:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.633282:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80314368, 80318464) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.633292:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80314368=0x4c98000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.634150:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.634157:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80318464 00000020:00010000:1.0:1789760790.634167:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.634170:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.634174:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.634177:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.634180:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.634188:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.634195:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.634240:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760790.634396:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 80314368<>80318464 00000100:00100000:1.0:1789760790.634444:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760790.634462:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503899392:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.645804:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986915 gen 2 00000008:00010000:1.0:1789760790.645843:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:1.0:1789760790.645850:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.645859:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.645881:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:1.0:1789760790.645908:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503899392:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760790.645999:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19608, 19608] 00000080:00200000:3.0:1789760790.646011:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760790.646016:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760790.646028:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80318464 i_blocks 156896 00000020:00200000:3.0:1789760790.646070:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760790.646081:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760790.646109:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80318464, count: 4096 00000080:00200000:3.0:1789760790.646115:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80318464, bytes: 4096 00000080:00200000:3.0:1789760790.646123:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760790.646131:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760790.646134:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80318464, 80322560) 0 0 00000080:00200000:3.0:1789760790.646138:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760790.646150:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19609, 19609] 00020000:00200000:3.0:1789760790.646157:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80318464 chunk: [80318464, 80322560) 80322560, 4096 00020000:00200000:3.0:1789760790.646161:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760790.646177:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19609, 19609] 00000008:00010000:3.0:1789760790.646192:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760790.646197:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760790.646206:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80318464, 80322560) append 0 append lockpos 0 00000080:00200000:3.0:1789760790.646218:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80318464=0x4c99000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760790.646263:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760790.646265:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80322560 00000020:00010000:3.0:1789760790.646272:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760790.646276:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.646279:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.646283:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760790.646285:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760790.646292:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760790.646298:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.646314:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760790.646401:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80318464<>80322560 00000100:00100000:3.0:1789760790.646439:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503899520:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.655263:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986916 gen 2 00000008:00010000:3.0:1789760790.655299:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760790.655306:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.655313:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.655341:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760790.655364:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503899520:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.656343:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760790.656359:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503899648:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.665254:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986917 gen 2 00000008:00010000:3.0:1789760790.665292:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760790.665298:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.665306:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.665324:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760790.665346:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503899648:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.666269:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503899776:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.686912:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986918 gen 2 00000080:00200000:3.0:1789760790.687233:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19611, 19611] 00000080:00200000:3.0:1789760790.687244:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760790.687248:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760790.687263:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80330752 i_blocks 156920 00000020:00200000:3.0:1789760790.687266:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760790.687273:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760790.687295:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80330752, count: 4096 00000080:00200000:3.0:1789760790.687300:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80330752, bytes: 4096 00000080:00200000:3.0:1789760790.687310:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760790.687316:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760790.687318:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80330752, 80334848) 0 0 00000080:00200000:3.0:1789760790.687322:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760790.687332:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19612, 19612] 00020000:00200000:3.0:1789760790.687337:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80330752 chunk: [80330752, 80334848) 80334848, 4096 00020000:00200000:3.0:1789760790.687340:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760790.687353:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19612, 19612] 00000008:00010000:3.0:1789760790.687365:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760790.687370:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760790.687377:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80330752, 80334848) append 0 append lockpos 0 00000080:00200000:3.0:1789760790.687387:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80330752=0x4c9c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760790.687422:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760790.687424:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80334848 00000020:00010000:3.0:1789760790.687431:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760790.687434:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.687436:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.687439:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760790.687442:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760790.687447:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760790.687451:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.687462:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760790.687532:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80330752<>80334848 00000100:00100000:3.0:1789760790.701175:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986919 gen 2 00000008:00010000:3.0:1789760790.701211:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760790.701217:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.701226:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.701250:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760790.701275:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503899904:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.701367:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19612, 19612] 00000080:00200000:1.0:1789760790.701381:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.701390:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.701402:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80334848 i_blocks 156928 00000020:00200000:1.0:1789760790.701406:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.701414:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.701445:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80334848, count: 4096 00000080:00200000:1.0:1789760790.701452:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80334848, bytes: 4096 00000080:00200000:1.0:1789760790.701461:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.701469:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.701471:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80334848, 80338944) 0 0 00000080:00200000:1.0:1789760790.701475:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.701487:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19613, 19613] 00020000:00200000:1.0:1789760790.701494:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80334848 chunk: [80334848, 80338944) 80338944, 4096 00020000:00200000:1.0:1789760790.701498:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.701512:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19613, 19613] 00000008:00010000:1.0:1789760790.701646:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.701651:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.701659:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80334848, 80338944) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.701670:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80334848=0x4c9d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.701732:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.701734:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80338944 00000020:00010000:1.0:1789760790.701741:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.701743:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.701746:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.701756:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.701759:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.701764:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.701769:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.701784:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760790.701867:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 80334848<>80338944 00000100:00100000:0.0:1789760790.701954:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503900032:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.716553:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986920 gen 2 00000008:00010000:0.0:1789760790.716610:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:0.0:1789760790.716616:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.716625:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.716656:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:0.0:1789760790.716681:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503900032:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.716810:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19613, 19613] 00000080:00200000:1.0:1789760790.716824:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.716830:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.716843:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80338944 i_blocks 156936 00000020:00200000:1.0:1789760790.716847:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.716856:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.716893:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80338944, count: 4096 00000080:00200000:1.0:1789760790.716900:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80338944, bytes: 4096 00000080:00200000:1.0:1789760790.716908:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.716917:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.716919:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80338944, 80343040) 0 0 00000080:00200000:1.0:1789760790.716923:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.716935:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19614, 19614] 00020000:00200000:1.0:1789760790.716942:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80338944 chunk: [80338944, 80343040) 80343040, 4096 00020000:00200000:1.0:1789760790.716946:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.716960:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19614, 19614] 00000008:00010000:1.0:1789760790.716975:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.716980:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.716990:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80338944, 80343040) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.717001:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80338944=0x4c9e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.717069:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.717072:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80343040 00000020:00010000:1.0:1789760790.717079:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.717082:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.717085:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.717088:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.717091:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.717097:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.717102:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.717126:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760790.717201:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 80338944<>80343040 00000100:00100000:0.0:1789760790.717991:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503900160:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.739523:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986921 gen 2 00000008:00010000:0.0:1789760790.739565:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58000 00000020:00200000:0.0:1789760790.739573:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.739582:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.739617:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58000 00000100:00100000:0.0:1789760790.739645:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503900160:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.739844:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19614, 19614] 00000080:00200000:1.0:1789760790.739857:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.739862:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.739874:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80343040 i_blocks 156944 00000020:00200000:1.0:1789760790.739878:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.739888:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.739913:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80343040, count: 4096 00000080:00200000:1.0:1789760790.739919:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80343040, bytes: 4096 00000080:00200000:1.0:1789760790.739930:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.739938:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.739940:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80343040, 80347136) 0 0 00000080:00200000:1.0:1789760790.739944:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.739957:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19615, 19615] 00020000:00200000:1.0:1789760790.739964:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80343040 chunk: [80343040, 80347136) 80347136, 4096 00020000:00200000:1.0:1789760790.739968:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.739984:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19615, 19615] 00000008:00010000:1.0:1789760790.739998:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.740003:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.740013:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80343040, 80347136) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.740024:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80343040=0x4c9f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.740092:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.740095:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80347136 00000020:00010000:1.0:1789760790.740102:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.740105:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.740108:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.740111:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.740113:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.740119:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.740125:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.740139:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760790.740218:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 80343040<>80347136 00000100:00100000:0.0:1789760790.740320:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503900288:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.753669:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986922 gen 2 00000008:00010000:0.0:1789760790.753706:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:0.0:1789760790.753712:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.753722:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.753741:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000100:00100000:0.0:1789760790.753766:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503900288:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.753860:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19615, 19615] 00000080:00200000:1.0:1789760790.753872:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.753877:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.753889:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80347136 i_blocks 156952 00000020:00200000:1.0:1789760790.753892:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.753899:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.753924:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80347136, count: 4096 00000080:00200000:1.0:1789760790.753930:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80347136, bytes: 4096 00000080:00200000:1.0:1789760790.753937:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.753942:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.753944:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80347136, 80351232) 0 0 00000080:00200000:1.0:1789760790.753948:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.753960:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19616, 19616] 00020000:00200000:1.0:1789760790.753966:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80347136 chunk: [80347136, 80351232) 80351232, 4096 00020000:00200000:1.0:1789760790.753970:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.753983:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19616, 19616] 00000008:00010000:1.0:1789760790.753996:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.754001:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.754009:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80347136, 80351232) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.754018:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80347136=0x4ca0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.754086:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.754089:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80351232 00000020:00010000:1.0:1789760790.754095:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.754099:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.754101:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.754104:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.754107:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.754113:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.754118:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.754132:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58780 00000008:00100000:1.0:1789760790.754206:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 80347136<>80351232 00000100:00100000:0.0:1789760790.754298:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503900416:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.776229:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986923 gen 2 00000008:00010000:0.0:1789760790.776264:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58780 00000020:00200000:0.0:1789760790.776271:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.776279:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.776298:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58780 00000100:00100000:0.0:1789760790.776328:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503900416:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.776364:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19616, 19616] 00000080:00200000:1.0:1789760790.776379:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.776384:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.776395:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80351232 i_blocks 156960 00000020:00200000:1.0:1789760790.776397:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.776406:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.776432:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80351232, count: 4096 00000080:00200000:1.0:1789760790.776438:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80351232, bytes: 4096 00000080:00200000:1.0:1789760790.776445:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.776451:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.776453:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80351232, 80355328) 0 0 00000080:00200000:1.0:1789760790.776456:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.776469:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19617, 19617] 00020000:00200000:1.0:1789760790.776476:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80351232 chunk: [80351232, 80355328) 80355328, 4096 00020000:00200000:1.0:1789760790.776479:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.776494:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19617, 19617] 00000008:00010000:1.0:1789760790.776507:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.776511:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.776520:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80351232, 80355328) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.776531:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80351232=0x4ca1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.776575:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.776577:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80355328 00000020:00010000:1.0:1789760790.776583:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.776586:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.776589:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.776592:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.776737:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.776746:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.776751:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.776765:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760790.776841:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 80351232<>80355328 00000100:00100000:0.0:1789760790.776992:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503900544:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.790281:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986924 gen 2 00000008:00010000:0.0:1789760790.790319:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:0.0:1789760790.790324:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.790333:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.790354:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:0.0:1789760790.790376:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503900544:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.790480:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19617, 19617] 00000080:00200000:1.0:1789760790.790492:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.790496:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.790508:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80355328 i_blocks 156968 00000020:00200000:1.0:1789760790.790512:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.790521:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.790547:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80355328, count: 4096 00000080:00200000:1.0:1789760790.790553:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80355328, bytes: 4096 00000080:00200000:1.0:1789760790.790561:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.790568:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.790570:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80355328, 80359424) 0 0 00000080:00200000:1.0:1789760790.790573:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.790586:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19618, 19618] 00020000:00200000:1.0:1789760790.790593:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80355328 chunk: [80355328, 80359424) 80359424, 4096 00020000:00200000:1.0:1789760790.791240:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.791257:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19618, 19618] 00000008:00010000:1.0:1789760790.791271:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.791276:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.791286:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80355328, 80359424) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.791297:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80355328=0x4ca2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.791395:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.791398:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80359424 00000020:00010000:1.0:1789760790.791405:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.791409:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.791411:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.791414:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.791425:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.791432:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.791438:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.791456:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760790.791535:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 80355328<>80359424 00000100:00100000:0.0:1789760790.792934:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503900672:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.810845:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986925 gen 2 00000008:00010000:0.0:1789760790.810883:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760790.810959:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.810970:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.810991:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760790.811016:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503900672:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.811058:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19618, 19618] 00000080:00200000:1.0:1789760790.811073:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.811078:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.811090:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80359424 i_blocks 156976 00000020:00200000:1.0:1789760790.811094:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.811104:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.811141:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80359424, count: 4096 00000080:00200000:1.0:1789760790.811148:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80359424, bytes: 4096 00000080:00200000:1.0:1789760790.811159:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.811166:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.811168:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80359424, 80363520) 0 0 00000080:00200000:1.0:1789760790.811172:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.811186:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19619, 19619] 00020000:00200000:1.0:1789760790.811193:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80359424 chunk: [80359424, 80363520) 80363520, 4096 00020000:00200000:1.0:1789760790.811197:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.811212:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19619, 19619] 00000008:00010000:1.0:1789760790.811227:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.811232:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.811241:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80359424, 80363520) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.811252:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80359424=0x4ca3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.811294:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.811297:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80363520 00000020:00010000:1.0:1789760790.811303:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.811307:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.811309:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.811312:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.811315:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.811322:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.811327:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.811341:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760790.811422:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 80359424<>80363520 00000100:00100000:0.0:1789760790.811482:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503900800:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.829800:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986926 gen 2 00000008:00010000:0.0:1789760790.829834:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760790.829841:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.829850:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.829871:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:0.0:1789760790.829894:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503900800:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.830170:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19619, 19619] 00000080:00200000:1.0:1789760790.830183:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.830189:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.830201:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80363520 i_blocks 156984 00000020:00200000:1.0:1789760790.830205:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.830213:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.830239:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80363520, count: 4096 00000080:00200000:1.0:1789760790.830245:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80363520, bytes: 4096 00000080:00200000:1.0:1789760790.830252:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.830260:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.830262:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80363520, 80367616) 0 0 00000080:00200000:1.0:1789760790.830265:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.830278:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19620, 19620] 00020000:00200000:1.0:1789760790.830285:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80363520 chunk: [80363520, 80367616) 80367616, 4096 00020000:00200000:1.0:1789760790.830289:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.830304:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19620, 19620] 00000008:00010000:1.0:1789760790.830751:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.830757:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.830768:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80363520, 80367616) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.830779:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80363520=0x4ca4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.830824:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.830827:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80367616 00000020:00010000:1.0:1789760790.830833:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.830836:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.830839:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.830842:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.830844:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.830850:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.830854:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.830869:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760790.830950:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 80363520<>80367616 00000100:00100000:0.0:1789760790.831023:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503900928:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.842299:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986927 gen 2 00000008:00010000:0.0:1789760790.842339:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760790.842346:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.842355:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.842365:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000080:00200000:1.0:1789760790.842415:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19620, 19620] 00000080:00200000:1.0:1789760790.842430:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.842436:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.842448:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80367616 i_blocks 156992 00000020:00200000:1.0:1789760790.842451:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.842460:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.842485:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80367616, count: 4096 00000080:00200000:1.0:1789760790.842492:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80367616, bytes: 4096 00000080:00200000:1.0:1789760790.842499:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.842506:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.842508:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80367616, 80371712) 0 0 00000080:00200000:1.0:1789760790.842512:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.842524:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19621, 19621] 00020000:00200000:1.0:1789760790.842536:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80367616 chunk: [80367616, 80371712) 80371712, 4096 00020000:00200000:1.0:1789760790.842541:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.842554:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19621, 19621] 00000008:00010000:1.0:1789760790.842567:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.842571:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.842580:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80367616, 80371712) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.842590:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80367616=0x4ca5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.842639:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.842642:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80371712 00000020:00010000:1.0:1789760790.842649:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.842653:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.842656:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.842659:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.842662:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.842668:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.842673:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1789760790.842745:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503900928:192.168.203.158@tcp:4:dd.0 00000008:00010000:0.0:1789760790.842784:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:0.0:1789760790.842876:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98700 - object 0x280000401:2 offset 80367616<>80371712 00000100:00100000:0.0:1789760790.842909:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760790.842921:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503901056:192.168.203.158@tcp:4:dd.0 00010000:00010000:3.0:1789760790.846301:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760790.860253:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986928 gen 2 00000008:00010000:0.0:1789760790.860301:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760790.860308:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.860320:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.860507:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000100:00100000:0.0:1789760790.860540:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503901056:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760790.864324:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19621, 19621] 00000080:00200000:1.0:1789760790.864354:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.864360:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.864375:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80371712 i_blocks 157000 00000020:00200000:1.0:1789760790.864379:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.864560:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.864586:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80371712, count: 4096 00000080:00200000:1.0:1789760790.864603:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80371712, bytes: 4096 00000080:00200000:1.0:1789760790.864613:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.864621:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760790.864623:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80371712, 80375808) 0 0 00000080:00200000:1.0:1789760790.864626:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.864641:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19622, 19622] 00020000:00200000:1.0:1789760790.864647:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80371712 chunk: [80371712, 80375808) 80375808, 4096 00020000:00200000:1.0:1789760790.864651:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.864667:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19622, 19622] 00000008:00010000:1.0:1789760790.864681:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.864686:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760790.864695:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80371712, 80375808) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.864707:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80371712=0x4ca6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.864766:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.864769:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80375808 00000020:00010000:1.0:1789760790.864777:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.864780:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.864783:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.864786:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.864789:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.864796:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.864802:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.864825:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760790.864909:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 80371712<>80375808 00000100:00100000:1.0:1789760790.864953:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760790.864973:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503901184:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.879081:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986929 gen 2 00000008:00010000:1.0:1789760790.879132:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:1.0:1789760790.879139:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.879148:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.879207:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:1.0:1789760790.879231:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503901184:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.884434:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503901312:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.894870:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986930 gen 2 00000008:00010000:3.0:1789760790.894904:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760790.894910:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.894919:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.894936:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760790.894958:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503901312:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.896250:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503901440:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.906350:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986931 gen 2 00000008:00010000:3.0:1789760790.906389:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760790.906396:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.906404:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.906421:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760790.906444:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503901440:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.907202:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503901568:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.915100:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986932 gen 2 00000008:00010000:3.0:1789760790.915145:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760790.915152:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.915161:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.915181:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760790.915206:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503901568:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.916025:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503901696:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.925785:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986933 gen 2 00000008:00010000:3.0:1789760790.925819:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760790.925825:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.925834:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.925853:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760790.925874:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503901696:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.944857:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986934 gen 2 00000008:00010000:3.0:1789760790.944890:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760790.944896:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.944904:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.944924:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760790.944946:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503901824:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.945032:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19627, 19627] 00000080:00200000:0.0:1789760790.945065:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.945069:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.945081:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80396288 i_blocks 157048 00000020:00200000:0.0:1789760790.945084:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.945092:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.945125:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80396288, count: 4096 00000080:00200000:0.0:1789760790.945131:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80396288, bytes: 4096 00000080:00200000:0.0:1789760790.945138:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.945145:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.945147:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80396288, 80400384) 0 0 00000080:00200000:0.0:1789760790.945150:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.945161:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19628, 19628] 00020000:00200000:0.0:1789760790.945167:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80396288 chunk: [80396288, 80400384) 80400384, 4096 00020000:00200000:0.0:1789760790.945171:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.945183:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19628, 19628] 00000008:00010000:0.0:1789760790.945195:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.945199:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.945206:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80396288, 80400384) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.945215:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80396288=0x4cac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.945254:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.945257:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80400384 00000020:00010000:0.0:1789760790.945262:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.945266:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.945268:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.945271:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.945274:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.945280:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.945285:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.945297:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760790.945377:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98700 - object 0x280000401:2 offset 80396288<>80400384 00000100:00100000:0.0:1789760790.945410:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760790.945422:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503901952:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760790.961484:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986935 gen 2 00000008:00010000:0.0:1789760790.961533:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:0.0:1789760790.961540:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.961550:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.961580:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:0.0:1789760790.961616:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503901952:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760790.961842:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19628, 19628] 00000080:00200000:3.0:1789760790.961856:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760790.961863:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760790.961875:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80400384 i_blocks 157056 00000020:00200000:3.0:1789760790.961878:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760790.961886:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760790.961909:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80400384, count: 4096 00000080:00200000:3.0:1789760790.961914:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80400384, bytes: 4096 00000080:00200000:3.0:1789760790.961924:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760790.961930:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760790.961932:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80400384, 80404480) 0 0 00000080:00200000:3.0:1789760790.961935:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760790.961946:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19629, 19629] 00020000:00200000:3.0:1789760790.961952:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80400384 chunk: [80400384, 80404480) 80404480, 4096 00020000:00200000:3.0:1789760790.961955:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760790.961969:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19629, 19629] 00000008:00010000:3.0:1789760790.961981:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760790.961985:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760790.961994:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80400384, 80404480) append 0 append lockpos 0 00000080:00200000:3.0:1789760790.962005:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80400384=0x4cad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760790.962086:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760790.962089:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80404480 00000020:00010000:3.0:1789760790.962095:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760790.962098:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.962163:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.962168:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760790.962170:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760790.962177:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760790.962182:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.962196:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760790.962266:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80400384<>80404480 00000100:00100000:3.0:1789760790.962301:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760790.962313:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503902080:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760790.970175:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986936 gen 2 00000008:00010000:3.0:1789760790.970229:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760790.970238:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.970248:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.970273:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760790.970305:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503902080:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.972190:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19629, 19629] 00000080:00200000:0.0:1789760790.972211:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.972216:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.972229:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80404480 i_blocks 157064 00000020:00200000:0.0:1789760790.972234:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.972245:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.972269:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80404480, count: 4096 00000080:00200000:0.0:1789760790.972275:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80404480, bytes: 4096 00000080:00200000:0.0:1789760790.972285:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.972291:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.972293:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80404480, 80408576) 0 0 00000080:00200000:0.0:1789760790.972297:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.972307:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19630, 19630] 00020000:00200000:0.0:1789760790.972314:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80404480 chunk: [80404480, 80408576) 80408576, 4096 00020000:00200000:0.0:1789760790.972318:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.972333:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19630, 19630] 00000008:00010000:0.0:1789760790.972347:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.972352:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.972362:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80404480, 80408576) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.972373:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80404480=0x4cae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.972427:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.972430:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80408576 00000020:00010000:0.0:1789760790.972438:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.972441:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.972444:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.972447:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.972449:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.972457:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.972462:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.972481:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760790.972570:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98700 - object 0x280000401:2 offset 80404480<>80408576 00000100:00100000:1.0:1789760790.972861:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503902208:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760790.987902:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986937 gen 2 00000008:00010000:1.0:1789760790.987950:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760790.987957:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.987968:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.988000:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760790.988028:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503902208:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760790.988176:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19630, 19630] 00000080:00200000:0.0:1789760790.988192:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.988199:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.988215:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80408576 i_blocks 157072 00000020:00200000:0.0:1789760790.988219:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.988229:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.988262:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80408576, count: 4096 00000080:00200000:0.0:1789760790.988269:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80408576, bytes: 4096 00000080:00200000:0.0:1789760790.988278:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.988286:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760790.988288:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80408576, 80412672) 0 0 00000080:00200000:0.0:1789760790.988292:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.988307:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19631, 19631] 00020000:00200000:0.0:1789760790.988315:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80408576 chunk: [80408576, 80412672) 80412672, 4096 00020000:00200000:0.0:1789760790.988319:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.988339:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19631, 19631] 00000008:00010000:0.0:1789760790.988370:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.988376:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760790.988388:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80408576, 80412672) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.988402:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80408576=0x4caf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.988467:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.988471:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80412672 00000020:00010000:0.0:1789760790.988479:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.988483:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.988486:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.988489:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.988492:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.988499:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.988506:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.988522:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760790.988638:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9bb80 - object 0x280000401:2 offset 80408576<>80412672 00000100:00100000:1.0:1789760790.988802:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503902336:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.008232:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986938 gen 2 00000008:00010000:1.0:1789760791.008290:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760791.008298:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.008310:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.008334:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760791.008365:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503902336:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.008480:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19631, 19631] 00000080:00200000:0.0:1789760791.008496:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.008502:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.008518:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80412672 i_blocks 157080 00000020:00200000:0.0:1789760791.008522:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.008532:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.008564:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80412672, count: 4096 00000080:00200000:0.0:1789760791.008572:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80412672, bytes: 4096 00000080:00200000:0.0:1789760791.008580:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.008589:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.008591:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80412672, 80416768) 0 0 00000080:00200000:0.0:1789760791.008605:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.008624:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19632, 19632] 00020000:00200000:0.0:1789760791.008633:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80412672 chunk: [80412672, 80416768) 80416768, 4096 00020000:00200000:0.0:1789760791.008638:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.008656:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19632, 19632] 00000008:00010000:0.0:1789760791.008674:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.008679:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.008691:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80412672, 80416768) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.008706:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80412672=0x4cb0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.008762:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.008765:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80416768 00000020:00010000:0.0:1789760791.008773:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.008776:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.008780:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.008783:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.008786:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.008793:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.008799:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.008813:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760791.008906:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99f80 - object 0x280000401:2 offset 80412672<>80416768 00000100:00100000:1.0:1789760791.009079:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503902464:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.017992:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986939 gen 2 00000008:00010000:1.0:1789760791.018078:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760791.018087:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.018098:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.018118:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760791.018150:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503902464:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.018251:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19632, 19632] 00000080:00200000:0.0:1789760791.018267:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.018273:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.018288:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80416768 i_blocks 157088 00000020:00200000:0.0:1789760791.018291:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.018301:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.018334:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80416768, count: 4096 00000080:00200000:0.0:1789760791.018340:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80416768, bytes: 4096 00000080:00200000:0.0:1789760791.018349:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.018358:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.018360:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80416768, 80420864) 0 0 00000080:00200000:0.0:1789760791.018365:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.018383:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19633, 19633] 00020000:00200000:0.0:1789760791.018491:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80416768 chunk: [80416768, 80420864) 80420864, 4096 00020000:00200000:0.0:1789760791.018496:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.018515:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19633, 19633] 00000008:00010000:0.0:1789760791.018533:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.018539:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.018549:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80416768, 80420864) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.018563:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80416768=0x4cb1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.018624:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.018627:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80420864 00000020:00010000:0.0:1789760791.018634:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.018637:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.018641:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.018644:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.018647:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.018654:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.018660:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.018675:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760791.018769:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99880 - object 0x280000401:2 offset 80416768<>80420864 00000100:00100000:1.0:1789760791.021571:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503902592:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.030229:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986940 gen 2 00000008:00010000:1.0:1789760791.030276:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760791.030285:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.030295:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.030320:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760791.030352:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503902592:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.030650:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19633, 19633] 00000080:00200000:0.0:1789760791.030667:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.030674:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.030691:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80420864 i_blocks 157096 00000020:00200000:0.0:1789760791.030695:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.030705:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.030742:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80420864, count: 4096 00000080:00200000:0.0:1789760791.030749:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80420864, bytes: 4096 00000080:00200000:0.0:1789760791.030760:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.030769:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.030772:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80420864, 80424960) 0 0 00000080:00200000:0.0:1789760791.030776:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.030792:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19634, 19634] 00020000:00200000:0.0:1789760791.030802:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80420864 chunk: [80420864, 80424960) 80424960, 4096 00020000:00200000:0.0:1789760791.030806:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.030826:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19634, 19634] 00000008:00010000:0.0:1789760791.030847:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.030852:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.030864:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80420864, 80424960) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.030877:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80420864=0x4cb2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.030936:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.030939:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80424960 00000020:00010000:0.0:1789760791.030947:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.030950:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.030953:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.030957:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.030960:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.030968:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.030976:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.030993:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760791.031143:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 80420864<>80424960 00000100:00100000:1.0:1789760791.031244:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503902720:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.046769:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986941 gen 2 00000008:00010000:1.0:1789760791.046816:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760791.046825:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.046837:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.046858:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760791.046890:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503902720:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.049782:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19634, 19634] 00000080:00200000:0.0:1789760791.049808:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.049815:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.049833:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80424960 i_blocks 157104 00000020:00200000:0.0:1789760791.049839:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.049856:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.049891:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80424960, count: 4096 00000080:00200000:0.0:1789760791.049899:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80424960, bytes: 4096 00000080:00200000:0.0:1789760791.049912:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.049921:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.049923:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80424960, 80429056) 0 0 00000080:00200000:0.0:1789760791.049928:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.049943:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19635, 19635] 00020000:00200000:0.0:1789760791.049951:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80424960 chunk: [80424960, 80429056) 80429056, 4096 00020000:00200000:0.0:1789760791.049956:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.049974:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19635, 19635] 00000008:00010000:0.0:1789760791.049993:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.049999:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.050011:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80424960, 80429056) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.050026:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80424960=0x4cb3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.050168:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.050171:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80429056 00000020:00010000:0.0:1789760791.050180:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.050183:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.050187:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.050191:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.050194:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.050203:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.050211:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.050225:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760791.050343:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98a80 - object 0x280000401:2 offset 80424960<>80429056 00000100:00100000:1.0:1789760791.050439:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503902848:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.066751:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986942 gen 2 00000008:00010000:1.0:1789760791.066798:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760791.066806:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.066819:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.066839:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760791.066868:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503902848:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.067514:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503902976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.079854:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986943 gen 2 00000008:00010000:3.0:1789760791.079898:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760791.079906:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.079916:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.079935:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760791.079963:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503902976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.080732:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903104:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.092106:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986944 gen 2 00000008:00010000:3.0:1789760791.092143:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760791.092149:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.092158:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.092179:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760791.092202:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903104:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.092880:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.092894:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.103517:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986945 gen 2 00000008:00010000:3.0:1789760791.103560:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760791.103567:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.103576:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.103603:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760791.103639:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.104668:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903360:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.114352:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986946 gen 2 00000008:00010000:3.0:1789760791.114389:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760791.114395:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.114403:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.114482:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760791.114519:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903360:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.115837:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.115853:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903488:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.127446:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986947 gen 2 00000008:00010000:3.0:1789760791.127484:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760791.127491:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.127499:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.129314:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760791.129348:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903488:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.150392:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986948 gen 2 00000008:00010000:3.0:1789760791.150427:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760791.150433:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.150441:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.150459:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760791.150481:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503903616:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.152196:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.152213:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903744:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.162557:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986949 gen 2 00000008:00010000:3.0:1789760791.162611:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760791.162617:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.162626:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.162836:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760791.162859:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903744:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.163547:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903872:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.180726:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986950 gen 2 00000008:00010000:3.0:1789760791.180762:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760791.180769:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.180777:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.180796:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760791.180820:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503903872:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.182333:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.182353:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904000:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.192358:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986951 gen 2 00000008:00010000:3.0:1789760791.192396:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760791.192402:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.192411:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.192428:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760791.192452:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904000:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.193150:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904128:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.202094:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986952 gen 2 00000008:00010000:3.0:1789760791.202133:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760791.202139:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.202148:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.202168:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760791.202193:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904128:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.202733:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503904256:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.213448:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986953 gen 2 00000008:00010000:3.0:1789760791.213483:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760791.213489:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.213498:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.213517:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760791.213542:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503904256:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.214072:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904384:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.229197:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986954 gen 2 00000008:00010000:3.0:1789760791.229233:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760791.229239:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.229247:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.229268:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760791.229292:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904384:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.229894:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503904512:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.250299:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986955 gen 2 00000008:00010000:3.0:1789760791.250347:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760791.250353:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.250361:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.250382:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760791.250406:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503904512:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.252636:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904640:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.273739:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986956 gen 2 00000008:00010000:3.0:1789760791.273775:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760791.273781:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.273789:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.273810:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760791.273833:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904640:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.274499:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.274513:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904768:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.291108:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986957 gen 2 00000008:00010000:3.0:1789760791.291142:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760791.291149:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.291158:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.291176:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760791.291199:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904768:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.291754:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904896:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.304787:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986958 gen 2 00000008:00010000:3.0:1789760791.304822:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760791.304828:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.304837:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.304864:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760791.304888:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503904896:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.305729:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503905024:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.327300:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986959 gen 2 00000008:00010000:3.0:1789760791.327347:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760791.327354:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.327362:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.327380:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760791.327403:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503905024:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.328641:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503905152:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.341212:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986960 gen 2 00000008:00010000:3.0:1789760791.341245:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760791.341250:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.341258:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.341280:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760791.341304:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503905152:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.342163:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503905280:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.355541:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986961 gen 2 00000008:00010000:3.0:1789760791.355580:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760791.355586:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.355604:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.355622:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760791.355654:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503905280:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.356894:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503905408:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.381972:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986962 gen 2 00000008:00010000:3.0:1789760791.382011:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760791.382017:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.382026:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.382070:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760791.382093:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503905408:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.382695:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503905536:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.392481:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986963 gen 2 00000008:00010000:3.0:1789760791.392525:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760791.392531:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.392539:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.392558:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760791.392581:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503905536:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.393467:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503905664:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.407846:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986964 gen 2 00000008:00010000:3.0:1789760791.407881:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760791.407900:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.407909:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.407925:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760791.407947:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503905664:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.408845:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503905792:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.419574:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986965 gen 2 00000008:00010000:3.0:1789760791.419620:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760791.419627:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.419810:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.419829:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760791.419853:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503905792:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760791.420432:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cd80 00000008:00100000:3.0:1789760791.420567:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80523264<>80527360 00000100:00100000:3.0:1789760791.420622:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503905920:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.433421:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986966 gen 2 00000008:00010000:3.0:1789760791.433454:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760791.433461:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.433469:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.433523:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760791.433548:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503905920:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.434294:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503906048:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.449616:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986967 gen 2 00000008:00010000:3.0:1789760791.449663:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760791.449670:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.449679:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.449696:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760791.449722:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503906048:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.450368:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503906176:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.458533:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986968 gen 2 00000008:00010000:3.0:1789760791.458568:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760791.458574:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.458582:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.458608:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760791.458637:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503906176:192.168.203.158@tcp:4:dd.0 00000008:00010000:3.0:1789760791.459827:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c780 00000008:00100000:3.0:1789760791.460086:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80535552<>80539648 00000100:00100000:3.0:1789760791.460126:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503906304:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.477905:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986969 gen 2 00000008:00010000:3.0:1789760791.477942:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760791.477948:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.477958:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.477978:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760791.478235:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503906304:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.481086:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503906432:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.493952:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986970 gen 2 00000008:00010000:3.0:1789760791.493999:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760791.494006:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.494014:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.494163:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760791.494188:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503906432:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.494911:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503906560:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.508556:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986971 gen 2 00000008:00010000:3.0:1789760791.508603:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760791.508610:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.508619:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.508647:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760791.508670:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503906560:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.508757:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19664, 19664] 00000080:00200000:0.0:1789760791.508770:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.508776:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.508787:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80547840 i_blocks 157344 00000020:00200000:0.0:1789760791.508790:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.508798:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.508823:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80547840, count: 4096 00000080:00200000:0.0:1789760791.508830:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80547840, bytes: 4096 00000080:00200000:0.0:1789760791.508837:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.508844:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.508846:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80547840, 80551936) 0 0 00000080:00200000:0.0:1789760791.508850:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.508862:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19665, 19665] 00020000:00200000:0.0:1789760791.508868:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80547840 chunk: [80547840, 80551936) 80551936, 4096 00020000:00200000:0.0:1789760791.508872:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.508886:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19665, 19665] 00000008:00010000:0.0:1789760791.508899:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.508904:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.508913:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80547840, 80551936) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.508924:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80547840=0x4cd1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.508967:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.508970:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80551936 00000020:00010000:0.0:1789760791.508977:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.508980:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.508982:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.508986:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.508988:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.508994:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.508999:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.509014:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760791.509119:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98000 - object 0x280000401:2 offset 80547840<>80551936 00000100:00100000:1.0:1789760791.509368:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760791.509382:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503906688:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.520409:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986972 gen 2 00000008:00010000:1.0:1789760791.520447:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760791.520453:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.520463:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.520474:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760791.520497:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503906688:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.520676:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19665, 19665] 00000080:00200000:0.0:1789760791.520689:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.520694:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.520706:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80551936 i_blocks 157352 00000020:00200000:0.0:1789760791.520709:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.520717:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.520743:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80551936, count: 4096 00000080:00200000:0.0:1789760791.520749:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80551936, bytes: 4096 00000080:00200000:0.0:1789760791.520756:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.520764:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.520766:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80551936, 80556032) 0 0 00000080:00200000:0.0:1789760791.520769:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.520782:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19666, 19666] 00020000:00200000:0.0:1789760791.520788:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80551936 chunk: [80551936, 80556032) 80556032, 4096 00020000:00200000:0.0:1789760791.520793:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.520807:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19666, 19666] 00000008:00010000:0.0:1789760791.520821:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.520825:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.520834:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80551936, 80556032) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.520844:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80551936=0x4cd2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.520886:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.520889:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80556032 00000020:00010000:0.0:1789760791.520895:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.520899:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.520901:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.520904:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.520907:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.520913:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.520919:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.520932:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760791.521006:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a680 - object 0x280000401:2 offset 80551936<>80556032 00000100:00100000:0.0:1789760791.521111:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503906816:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760791.533972:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986973 gen 2 00000008:00010000:0.0:1789760791.534009:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:0.0:1789760791.534016:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.534025:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.534074:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:0.0:1789760791.534097:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503906816:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760791.534281:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19666, 19666] 00000080:00200000:3.0:1789760791.534294:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760791.534300:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760791.534312:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80556032 i_blocks 157360 00000020:00200000:3.0:1789760791.534315:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760791.534332:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760791.534360:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80556032, count: 4096 00000080:00200000:3.0:1789760791.534366:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80556032, bytes: 4096 00000080:00200000:3.0:1789760791.534374:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760791.534382:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760791.534384:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80556032, 80560128) 0 0 00000080:00200000:3.0:1789760791.534388:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760791.534401:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19667, 19667] 00020000:00200000:3.0:1789760791.534407:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80556032 chunk: [80556032, 80560128) 80560128, 4096 00020000:00200000:3.0:1789760791.534411:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760791.534425:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19667, 19667] 00000008:00010000:3.0:1789760791.534438:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760791.534443:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760791.534451:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80556032, 80560128) append 0 append lockpos 0 00000080:00200000:3.0:1789760791.534463:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80556032=0x4cd3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760791.534503:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760791.534506:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80560128 00000020:00010000:3.0:1789760791.534512:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760791.534516:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.534518:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.534521:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760791.534524:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760791.534531:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760791.534535:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760791.534547:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760791.534626:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80556032<>80560128 00000100:00100000:3.0:1789760791.534780:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760791.534794:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503906944:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.548763:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986974 gen 2 00000008:00010000:3.0:1789760791.548804:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760791.548811:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.548818:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.548841:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760791.548864:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503906944:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.548962:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19667, 19667] 00000080:00200000:0.0:1789760791.548974:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.548979:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.548991:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80560128 i_blocks 157368 00000020:00200000:0.0:1789760791.548994:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.549002:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.549114:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80560128, count: 4096 00000080:00200000:0.0:1789760791.549122:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80560128, bytes: 4096 00000080:00200000:0.0:1789760791.549129:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.549137:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.549140:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80560128, 80564224) 0 0 00000080:00200000:0.0:1789760791.549144:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.549156:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19668, 19668] 00020000:00200000:0.0:1789760791.549163:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80560128 chunk: [80560128, 80564224) 80564224, 4096 00020000:00200000:0.0:1789760791.549167:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.549182:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19668, 19668] 00000008:00010000:0.0:1789760791.549261:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.549267:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.549276:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80560128, 80564224) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.549287:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80560128=0x4cd4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.549327:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.549330:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80564224 00000020:00010000:0.0:1789760791.549336:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.549339:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.549463:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.549466:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.549468:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.549476:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.549481:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.549496:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760791.549568:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a680 - object 0x280000401:2 offset 80560128<>80564224 00000100:00100000:1.0:1789760791.549655:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503907072:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.566354:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986975 gen 2 00000008:00010000:1.0:1789760791.566392:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760791.566398:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.566406:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.566427:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760791.566452:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503907072:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.566666:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19668, 19668] 00000080:00200000:0.0:1789760791.566679:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.566684:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.566696:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80564224 i_blocks 157376 00000020:00200000:0.0:1789760791.566699:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.566707:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.566734:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80564224, count: 4096 00000080:00200000:0.0:1789760791.566740:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80564224, bytes: 4096 00000080:00200000:0.0:1789760791.566747:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.566754:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.566756:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80564224, 80568320) 0 0 00000080:00200000:0.0:1789760791.566760:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.566772:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19669, 19669] 00020000:00200000:0.0:1789760791.566780:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80564224 chunk: [80564224, 80568320) 80568320, 4096 00020000:00200000:0.0:1789760791.566783:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.566797:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19669, 19669] 00000008:00010000:0.0:1789760791.566810:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.566815:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.566824:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80564224, 80568320) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.566834:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80564224=0x4cd5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.566875:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.566879:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80568320 00000020:00010000:0.0:1789760791.566885:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.566888:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.566891:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.566894:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.566896:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.566902:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.566909:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.566922:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760791.566992:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b480 - object 0x280000401:2 offset 80564224<>80568320 00000100:00100000:1.0:1789760791.567220:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907200:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.580687:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986976 gen 2 00000008:00010000:1.0:1789760791.580723:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760791.580729:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.580738:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.580760:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760791.580785:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907200:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.581785:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19669, 19669] 00000080:00200000:0.0:1789760791.581801:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.581806:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.581818:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80568320 i_blocks 157384 00000020:00200000:0.0:1789760791.581822:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.581831:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.581855:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80568320, count: 4096 00000080:00200000:0.0:1789760791.581861:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80568320, bytes: 4096 00000080:00200000:0.0:1789760791.581870:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.581879:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.581881:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80568320, 80572416) 0 0 00000080:00200000:0.0:1789760791.581885:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.581896:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19670, 19670] 00020000:00200000:0.0:1789760791.581903:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80568320 chunk: [80568320, 80572416) 80572416, 4096 00020000:00200000:0.0:1789760791.581907:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.581919:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19670, 19670] 00000008:00010000:0.0:1789760791.581932:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.581937:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.581944:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80568320, 80572416) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.581955:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80568320=0x4cd6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.582000:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.582003:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80572416 00000020:00010000:0.0:1789760791.582009:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.582012:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.582015:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.582018:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.582020:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.582074:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.582080:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.582096:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760791.582170:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b800 - object 0x280000401:2 offset 80568320<>80572416 00000100:00100000:1.0:1789760791.582363:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760791.582380:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907328:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.600655:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986977 gen 2 00000008:00010000:1.0:1789760791.600690:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760791.600695:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.600705:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.600723:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760791.600744:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907328:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.600890:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19670, 19670] 00000080:00200000:0.0:1789760791.600903:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.600908:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.600920:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80572416 i_blocks 157392 00000020:00200000:0.0:1789760791.600923:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.600930:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.600955:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80572416, count: 4096 00000080:00200000:0.0:1789760791.600960:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80572416, bytes: 4096 00000080:00200000:0.0:1789760791.600967:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.600974:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.600976:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80572416, 80576512) 0 0 00000080:00200000:0.0:1789760791.600981:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.600992:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19671, 19671] 00020000:00200000:0.0:1789760791.600999:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80572416 chunk: [80572416, 80576512) 80576512, 4096 00020000:00200000:0.0:1789760791.601002:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.601018:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19671, 19671] 00000008:00010000:0.0:1789760791.601072:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.601077:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.601086:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80572416, 80576512) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.601096:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80572416=0x4cd7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.601138:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.601141:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80576512 00000020:00010000:0.0:1789760791.601148:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.601151:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.601155:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.601158:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.601160:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.601167:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.601173:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.601186:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760791.601272:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9aa00 - object 0x280000401:2 offset 80572416<>80576512 00000100:00100000:1.0:1789760791.601498:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907456:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.615081:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986978 gen 2 00000008:00010000:1.0:1789760791.615120:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760791.615126:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.615136:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.615157:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760791.615183:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907456:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.615416:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19671, 19671] 00000080:00200000:0.0:1789760791.615428:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.615433:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.615446:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80576512 i_blocks 157400 00000020:00200000:0.0:1789760791.615449:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.615456:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.615482:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80576512, count: 4096 00000080:00200000:0.0:1789760791.615488:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80576512, bytes: 4096 00000080:00200000:0.0:1789760791.615496:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.615503:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.615506:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80576512, 80580608) 0 0 00000080:00200000:0.0:1789760791.615510:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.615522:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19672, 19672] 00020000:00200000:0.0:1789760791.615528:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80576512 chunk: [80576512, 80580608) 80580608, 4096 00020000:00200000:0.0:1789760791.615532:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.615546:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19672, 19672] 00000008:00010000:0.0:1789760791.615559:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.615563:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.615572:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80576512, 80580608) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.615582:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80576512=0x4cd8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.615636:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.615638:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80580608 00000020:00010000:0.0:1789760791.615645:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.615649:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.615651:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.615654:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.615656:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.615663:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.615668:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.615683:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760791.615752:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b100 - object 0x280000401:2 offset 80576512<>80580608 00000100:00100000:1.0:1789760791.616345:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760791.616364:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907584:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.628224:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986979 gen 2 00000008:00010000:1.0:1789760791.628281:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760791.628292:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.628308:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.628399:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760791.628433:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907584:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.628521:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19672, 19672] 00000080:00200000:0.0:1789760791.628603:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.628610:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.628626:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80580608 i_blocks 157408 00000020:00200000:0.0:1789760791.628691:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.628771:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.628810:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80580608, count: 4096 00000080:00200000:0.0:1789760791.628818:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80580608, bytes: 4096 00000080:00200000:0.0:1789760791.628830:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.628840:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.628844:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80580608, 80584704) 0 0 00000080:00200000:0.0:1789760791.628848:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.628866:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19673, 19673] 00020000:00200000:0.0:1789760791.628876:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80580608 chunk: [80580608, 80584704) 80584704, 4096 00020000:00200000:0.0:1789760791.628881:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.628902:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19673, 19673] 00000008:00010000:0.0:1789760791.628920:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.628926:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.628937:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80580608, 80584704) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.628950:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80580608=0x4cd9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.629009:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.629012:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80584704 00000020:00010000:0.0:1789760791.629020:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.629024:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.629028:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.629031:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.629120:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.629130:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.629138:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.629156:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760791.629262:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9ad80 - object 0x280000401:2 offset 80580608<>80584704 00000100:00100000:1.0:1789760791.629326:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907712:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.645955:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986980 gen 2 00000008:00010000:1.0:1789760791.646010:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760791.646019:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.646031:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.646154:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760791.646197:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907712:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.646293:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19673, 19673] 00000080:00200000:0.0:1789760791.646311:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.646316:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.646333:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80584704 i_blocks 157416 00000020:00200000:0.0:1789760791.646337:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.646347:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.646396:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80584704, count: 4096 00000080:00200000:0.0:1789760791.646404:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80584704, bytes: 4096 00000080:00200000:0.0:1789760791.646414:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.646424:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.646427:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80584704, 80588800) 0 0 00000080:00200000:0.0:1789760791.646432:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.646448:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19674, 19674] 00020000:00200000:0.0:1789760791.646456:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80584704 chunk: [80584704, 80588800) 80588800, 4096 00020000:00200000:0.0:1789760791.646461:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.646480:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19674, 19674] 00000008:00010000:0.0:1789760791.646787:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.646794:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.646807:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80584704, 80588800) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.646820:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80584704=0x4cda000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.646884:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.646887:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80588800 00000020:00010000:0.0:1789760791.646896:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.646899:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.646902:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.646906:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.646909:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.646918:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.646924:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.646944:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760791.647096:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98380 - object 0x280000401:2 offset 80584704<>80588800 00000100:00100000:1.0:1789760791.647194:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760791.647216:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907840:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.659331:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986981 gen 2 00000008:00010000:1.0:1789760791.659381:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760791.659390:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.659403:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.659426:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760791.659459:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503907840:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.661177:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19674, 19674] 00000080:00200000:0.0:1789760791.661200:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.661206:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.661224:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80588800 i_blocks 157424 00000020:00200000:0.0:1789760791.661230:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.661242:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.661278:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80588800, count: 4096 00000080:00200000:0.0:1789760791.661285:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80588800, bytes: 4096 00000080:00200000:0.0:1789760791.661298:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.661307:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.661311:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80588800, 80592896) 0 0 00000080:00200000:0.0:1789760791.661324:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.661342:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19675, 19675] 00020000:00200000:0.0:1789760791.661351:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80588800 chunk: [80588800, 80592896) 80592896, 4096 00020000:00200000:0.0:1789760791.661355:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.661377:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19675, 19675] 00000008:00010000:0.0:1789760791.661397:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.661403:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.661415:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80588800, 80592896) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.661429:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80588800=0x4cdb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.662210:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.662217:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80592896 00000020:00010000:0.0:1789760791.662228:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.662232:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.662236:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.662239:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.662242:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.662761:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.662774:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.662809:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760791.662925:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99c00 - object 0x280000401:2 offset 80588800<>80592896 00000100:00100000:0.0:1789760791.662980:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760791.663085:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503907968:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760791.674600:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986982 gen 2 00000008:00010000:0.0:1789760791.674706:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:0.0:1789760791.674715:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.674726:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.674749:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:0.0:1789760791.674781:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503907968:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760791.674908:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19675, 19675] 00000080:00200000:1.0:1789760791.674925:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.674930:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.674944:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80592896 i_blocks 157432 00000020:00200000:1.0:1789760791.674946:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.674955:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.674986:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80592896, count: 4096 00000080:00200000:1.0:1789760791.674991:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80592896, bytes: 4096 00000080:00200000:1.0:1789760791.675071:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.675080:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760791.675081:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80592896, 80596992) 0 0 00000080:00200000:1.0:1789760791.675085:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.675101:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19676, 19676] 00020000:00200000:1.0:1789760791.675109:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80592896 chunk: [80592896, 80596992) 80596992, 4096 00020000:00200000:1.0:1789760791.675113:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.675130:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19676, 19676] 00000008:00010000:1.0:1789760791.675145:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.675150:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760791.675159:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80592896, 80596992) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.675171:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80592896=0x4cdc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.675219:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.675221:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80596992 00000020:00010000:1.0:1789760791.675236:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.675239:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.675242:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.675245:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.675247:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.675253:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.675258:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.675271:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760791.675381:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 80592896<>80596992 00000100:00100000:1.0:1789760791.675424:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760791.675446:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908096:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.691526:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986983 gen 2 00000008:00010000:1.0:1789760791.691577:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:1.0:1789760791.691586:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.691613:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.691708:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:1.0:1789760791.691741:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908096:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760791.691894:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19676, 19676] 00000080:00200000:3.0:1789760791.691912:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760791.691920:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760791.691937:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80596992 i_blocks 157440 00000020:00200000:3.0:1789760791.691940:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760791.691951:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760791.691987:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80596992, count: 4096 00000080:00200000:3.0:1789760791.691994:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80596992, bytes: 4096 00000080:00200000:3.0:1789760791.692005:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760791.692014:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760791.692016:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80596992, 80601088) 0 0 00000080:00200000:3.0:1789760791.692021:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760791.692079:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19677, 19677] 00020000:00200000:3.0:1789760791.692088:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80596992 chunk: [80596992, 80601088) 80601088, 4096 00020000:00200000:3.0:1789760791.692092:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760791.692114:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19677, 19677] 00000008:00010000:3.0:1789760791.692470:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760791.692478:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760791.692493:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80596992, 80601088) append 0 append lockpos 0 00000080:00200000:3.0:1789760791.692509:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80596992=0x4cdd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760791.692570:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760791.692573:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80601088 00000020:00010000:3.0:1789760791.692583:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760791.692586:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.692600:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.692604:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760791.692606:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760791.692615:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760791.692622:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760791.692640:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760791.693014:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80596992<>80601088 00000100:00100000:3.0:1789760791.704568:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986984 gen 2 00000008:00010000:3.0:1789760791.704634:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:3.0:1789760791.704642:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.704658:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.704688:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:3.0:1789760791.704718:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503908224:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760791.704809:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19677, 19677] 00000080:00200000:1.0:1789760791.704825:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.704830:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.704846:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80601088 i_blocks 157448 00000020:00200000:1.0:1789760791.704849:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.704860:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.704894:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80601088, count: 4096 00000080:00200000:1.0:1789760791.704901:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80601088, bytes: 4096 00000080:00200000:1.0:1789760791.704910:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.704919:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760791.704921:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80601088, 80605184) 0 0 00000080:00200000:1.0:1789760791.704926:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.704942:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19678, 19678] 00020000:00200000:1.0:1789760791.704951:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80601088 chunk: [80601088, 80605184) 80605184, 4096 00020000:00200000:1.0:1789760791.704956:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.704989:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19678, 19678] 00000008:00010000:1.0:1789760791.705009:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.705016:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760791.705026:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80601088, 80605184) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.705073:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80601088=0x4cde000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.705133:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.705135:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80605184 00000020:00010000:1.0:1789760791.705144:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.705147:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.705150:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.705154:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.705156:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.705164:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.705170:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.705186:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760791.705288:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 80601088<>80605184 00000100:00100000:1.0:1789760791.707276:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908352:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.722935:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986985 gen 2 00000008:00010000:1.0:1789760791.722991:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:1.0:1789760791.723001:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.723013:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.723074:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000080:00200000:3.0:1789760791.723268:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19678, 19678] 00000080:00200000:3.0:1789760791.723287:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760791.723293:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760791.723311:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80605184 i_blocks 157456 00000020:00200000:3.0:1789760791.723315:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760791.723326:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760791.723362:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80605184, count: 4096 00000080:00200000:3.0:1789760791.723370:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80605184, bytes: 4096 00000080:00200000:3.0:1789760791.723381:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760791.723391:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760791.723394:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80605184, 80609280) 0 0 00000080:00200000:3.0:1789760791.723398:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760791.723414:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19679, 19679] 00020000:00200000:3.0:1789760791.723424:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80605184 chunk: [80605184, 80609280) 80609280, 4096 00020000:00200000:3.0:1789760791.723429:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760791.723449:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19679, 19679] 00000008:00010000:3.0:1789760791.723467:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760791.723473:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760791.723484:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80605184, 80609280) append 0 append lockpos 0 00000080:00200000:3.0:1789760791.723498:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80605184=0x4cdf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760791.723546:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760791.723549:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80609280 00000020:00010000:3.0:1789760791.723563:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760791.723567:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.723569:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.723572:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760791.723575:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760791.723581:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760791.723599:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760791.723656:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908352:192.168.203.158@tcp:4:dd.0 00000008:00010000:1.0:1789760791.723840:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:1.0:1789760791.723940:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 80605184<>80609280 00000100:00100000:1.0:1789760791.723982:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760791.723996:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908480:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.748389:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986986 gen 2 00000008:00010000:1.0:1789760791.748448:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:1.0:1789760791.748457:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.748470:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.748498:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:1.0:1789760791.748534:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908480:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760791.748780:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19679, 19679] 00000080:00200000:3.0:1789760791.748799:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760791.748806:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760791.748824:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80609280 i_blocks 157464 00000020:00200000:3.0:1789760791.748827:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760791.748838:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760791.748875:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80609280, count: 4096 00000080:00200000:3.0:1789760791.748883:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80609280, bytes: 4096 00000080:00200000:3.0:1789760791.748894:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760791.748904:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760791.748906:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80609280, 80613376) 0 0 00000080:00200000:3.0:1789760791.748912:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760791.748930:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19680, 19680] 00020000:00200000:3.0:1789760791.748940:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80609280 chunk: [80609280, 80613376) 80613376, 4096 00020000:00200000:3.0:1789760791.748945:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760791.748965:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19680, 19680] 00000008:00010000:3.0:1789760791.748984:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760791.748991:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760791.749002:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80609280, 80613376) append 0 append lockpos 0 00000080:00200000:3.0:1789760791.749016:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80609280=0x4ce0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760791.749117:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760791.749121:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80613376 00000020:00010000:3.0:1789760791.749128:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760791.749132:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.749135:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.749138:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760791.749141:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760791.749148:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760791.749155:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760791.749174:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1f00 00000008:00100000:3.0:1789760791.749278:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80609280<>80613376 00000100:00100000:3.0:1789760791.761806:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986987 gen 2 00000008:00010000:3.0:1789760791.761862:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1f00 00000020:00200000:3.0:1789760791.761872:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.761886:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.761913:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1f00 00000100:00100000:3.0:1789760791.761944:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503908608:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760791.763210:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19680, 19680] 00000080:00200000:1.0:1789760791.763233:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.763239:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.763257:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80613376 i_blocks 157472 00000020:00200000:1.0:1789760791.763263:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.763277:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.763313:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80613376, count: 4096 00000080:00200000:1.0:1789760791.763320:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80613376, bytes: 4096 00000080:00200000:1.0:1789760791.763333:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.763342:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760791.763356:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80613376, 80617472) 0 0 00000080:00200000:1.0:1789760791.763360:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.763378:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19681, 19681] 00020000:00200000:1.0:1789760791.763386:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80613376 chunk: [80613376, 80617472) 80617472, 4096 00020000:00200000:1.0:1789760791.763390:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.763411:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19681, 19681] 00000008:00010000:1.0:1789760791.763440:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.763447:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760791.763459:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80613376, 80617472) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.763473:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80613376=0x4ce1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.763538:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.763541:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80617472 00000020:00010000:1.0:1789760791.763548:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.763552:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.763554:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.763560:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.763563:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.763571:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.763579:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.763612:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760791.763719:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 80613376<>80617472 00000100:00100000:0.0:1789760791.763968:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908736:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760791.776268:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986988 gen 2 00000008:00010000:0.0:1789760791.776319:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760791.776327:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.776340:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.776377:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:0.0:1789760791.776410:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908736:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760791.776837:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19681, 19681] 00000080:00200000:1.0:1789760791.776854:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.776861:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.776878:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80617472 i_blocks 157480 00000020:00200000:1.0:1789760791.776882:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.776892:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.776930:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80617472, count: 4096 00000080:00200000:1.0:1789760791.776938:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80617472, bytes: 4096 00000080:00200000:1.0:1789760791.776948:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.776958:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760791.776960:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80617472, 80621568) 0 0 00000080:00200000:1.0:1789760791.776965:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.776982:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19682, 19682] 00020000:00200000:1.0:1789760791.776991:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80617472 chunk: [80617472, 80621568) 80621568, 4096 00020000:00200000:1.0:1789760791.776997:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.777017:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19682, 19682] 00000008:00010000:1.0:1789760791.777069:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.777076:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760791.777086:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80617472, 80621568) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.777105:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80617472=0x4ce2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.777165:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.777169:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80621568 00000020:00010000:1.0:1789760791.777177:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.777180:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.777183:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.777188:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.777191:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.777198:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.777205:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.777223:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760791.777332:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 80617472<>80621568 00000100:00100000:0.0:1789760791.778008:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760791.778066:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908864:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760791.788986:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986989 gen 2 00000008:00010000:0.0:1789760791.789081:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760791.789089:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.789101:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.789113:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760791.789144:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908864:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760791.789263:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19682, 19682] 00000080:00200000:1.0:1789760791.789280:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.789286:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.789303:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80621568 i_blocks 157488 00000020:00200000:1.0:1789760791.789307:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.789317:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.789353:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80621568, count: 4096 00000080:00200000:1.0:1789760791.789361:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80621568, bytes: 4096 00000080:00200000:1.0:1789760791.789382:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.789393:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760791.789395:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80621568, 80625664) 0 0 00000080:00200000:1.0:1789760791.789399:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.789417:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19683, 19683] 00020000:00200000:1.0:1789760791.789426:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80621568 chunk: [80621568, 80625664) 80625664, 4096 00020000:00200000:1.0:1789760791.789431:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.789451:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19683, 19683] 00000008:00010000:1.0:1789760791.789472:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.789479:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760791.789490:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80621568, 80625664) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.789505:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80621568=0x4ce3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.789579:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.789582:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80625664 00000020:00010000:1.0:1789760791.789605:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.789609:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.789612:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.789617:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.789620:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.789628:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.789635:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.789652:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760791.789757:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 80621568<>80625664 00000100:00100000:0.0:1789760791.789927:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908992:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760791.804117:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986990 gen 2 00000008:00010000:0.0:1789760791.804169:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:0.0:1789760791.804177:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.804189:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.804213:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:0.0:1789760791.804246:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503908992:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760791.804371:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19683, 19683] 00000080:00200000:1.0:1789760791.804387:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.804393:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.804409:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80625664 i_blocks 157496 00000020:00200000:1.0:1789760791.804414:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.804425:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.804462:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80625664, count: 4096 00000080:00200000:1.0:1789760791.804469:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80625664, bytes: 4096 00000080:00200000:1.0:1789760791.804478:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.804487:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760791.804489:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80625664, 80629760) 0 0 00000080:00200000:1.0:1789760791.804494:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.804512:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19684, 19684] 00020000:00200000:1.0:1789760791.804521:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80625664 chunk: [80625664, 80629760) 80629760, 4096 00020000:00200000:1.0:1789760791.804526:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.804546:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19684, 19684] 00000008:00010000:1.0:1789760791.804563:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.804569:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760791.804581:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80625664, 80629760) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.804604:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80625664=0x4ce4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.804666:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.804669:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80629760 00000020:00010000:1.0:1789760791.804678:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.804682:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.804685:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.804689:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.804692:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.804700:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.804709:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.804725:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58780 00000008:00100000:1.0:1789760791.804825:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 80625664<>80629760 00000100:00100000:0.0:1789760791.804892:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503909120:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.817681:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986991 gen 2 00000008:00010000:1.0:1789760791.817734:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58780 00000020:00200000:1.0:1789760791.817742:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.817754:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.817784:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58780 00000100:00100000:1.0:1789760791.817827:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503909120:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760791.817954:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19684, 19684] 00000080:00200000:3.0:1789760791.817970:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760791.817976:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760791.817992:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80629760 i_blocks 157504 00000020:00200000:3.0:1789760791.817996:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760791.818006:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760791.818080:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80629760, count: 4096 00000080:00200000:3.0:1789760791.818088:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80629760, bytes: 4096 00000080:00200000:3.0:1789760791.818100:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760791.818109:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760791.818111:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80629760, 80633856) 0 0 00000080:00200000:3.0:1789760791.818115:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760791.818133:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19685, 19685] 00020000:00200000:3.0:1789760791.818141:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80629760 chunk: [80629760, 80633856) 80633856, 4096 00020000:00200000:3.0:1789760791.818146:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760791.818166:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19685, 19685] 00000008:00010000:3.0:1789760791.818196:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760791.818203:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760791.818215:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80629760, 80633856) append 0 append lockpos 0 00000080:00200000:3.0:1789760791.818229:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80629760=0x4ce5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760791.818288:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760791.818291:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80633856 00000020:00010000:3.0:1789760791.818301:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760791.818305:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.818308:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.818312:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760791.818315:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760791.818323:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760791.818329:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760791.818466:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1f00 00000008:00100000:3.0:1789760791.818568:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80629760<>80633856 00000100:00100000:3.0:1789760791.818622:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.818641:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503909248:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.843394:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986992 gen 2 00000008:00010000:3.0:1789760791.843452:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1f00 00000020:00200000:3.0:1789760791.843461:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.843474:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.843511:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1f00 00000100:00100000:3.0:1789760791.843543:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503909248:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760791.843693:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19685, 19685] 00000080:00200000:1.0:1789760791.843710:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.843716:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.843732:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80633856 i_blocks 157512 00000020:00200000:1.0:1789760791.843736:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.843746:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.843781:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80633856, count: 4096 00000080:00200000:1.0:1789760791.843788:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80633856, bytes: 4096 00000080:00200000:1.0:1789760791.843801:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.843810:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760791.843812:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80633856, 80637952) 0 0 00000080:00200000:1.0:1789760791.843817:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.843834:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19686, 19686] 00020000:00200000:1.0:1789760791.843843:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80633856 chunk: [80633856, 80637952) 80637952, 4096 00020000:00200000:1.0:1789760791.843847:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.843867:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19686, 19686] 00000008:00010000:1.0:1789760791.843883:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.843889:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760791.843901:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80633856, 80637952) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.843916:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80633856=0x4ce6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.843972:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.843975:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80637952 00000020:00010000:1.0:1789760791.843983:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.843987:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.843990:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631230 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.844718:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.844721:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631230 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.844732:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.844741:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.844767:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58780 00000008:00100000:1.0:1789760791.844879:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 80633856<>80637952 00000100:00100000:0.0:1789760791.844955:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503909376:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760791.857436:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986993 gen 2 00000008:00010000:0.0:1789760791.857487:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58780 00000020:00200000:0.0:1789760791.857506:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.857518:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.857539:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58780 00000100:00100000:0.0:1789760791.857570:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503909376:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760791.857781:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19686, 19686] 00000080:00200000:1.0:1789760791.857798:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.857805:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.857823:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80637952 i_blocks 157520 00000020:00200000:1.0:1789760791.857826:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.857836:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.857870:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80637952, count: 4096 00000080:00200000:1.0:1789760791.857877:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80637952, bytes: 4096 00000080:00200000:1.0:1789760791.857889:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.857898:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760791.857900:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80637952, 80642048) 0 0 00000080:00200000:1.0:1789760791.857904:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.857921:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19687, 19687] 00020000:00200000:1.0:1789760791.857930:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80637952 chunk: [80637952, 80642048) 80642048, 4096 00020000:00200000:1.0:1789760791.857935:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.857955:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19687, 19687] 00000008:00010000:1.0:1789760791.857975:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.857981:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631230, osc lock ffff9018d181eab0, flags 0x0 00000080:00200000:1.0:1789760791.858004:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80637952, 80642048) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.858020:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80637952=0x4ce7000, pages 1 (max 131072), lockless, parallel 00000100:00100000:3.0:1789760791.884737:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986994 gen 2 00000008:00010000:3.0:1789760791.884791:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760791.884800:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.884812:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.884835:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760791.885349:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503909504:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.886314:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.886334:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503909632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.898426:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986995 gen 2 00000008:00010000:3.0:1789760791.898478:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760791.898487:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.898508:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.898528:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760791.898560:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503909632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.899533:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503909760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.910156:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986996 gen 2 00000008:00010000:3.0:1789760791.910208:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760791.910217:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.910878:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.910953:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760791.910988:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503909760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.913633:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503909888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.924930:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986997 gen 2 00000008:00010000:3.0:1789760791.924983:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760791.924992:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.925003:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.925022:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760791.925154:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503909888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.926094:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760791.926115:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503910016:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.941359:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986998 gen 2 00000008:00010000:3.0:1789760791.941423:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760791.941432:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.941443:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.941464:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760791.941495:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503910016:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.943025:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.943145:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503910144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.958138:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294986999 gen 2 00000008:00010000:3.0:1789760791.958192:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760791.958200:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.958212:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.958370:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760791.958401:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503910144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.963393:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503910272:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760791.974215:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987000 gen 2 00000008:00010000:3.0:1789760791.974262:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760791.974270:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.974279:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.974313:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760791.974341:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503910272:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.974514:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19693, 19693] 00000080:00200000:0.0:1789760791.974531:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.974537:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.974554:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80666624 i_blocks 157576 00000020:00200000:0.0:1789760791.974558:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.974569:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.974623:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80666624, count: 4096 00000080:00200000:0.0:1789760791.974640:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80666624, bytes: 4096 00000080:00200000:0.0:1789760791.974651:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.974660:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.974663:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80666624, 80670720) 0 0 00000080:00200000:0.0:1789760791.974667:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.974685:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19694, 19694] 00020000:00200000:0.0:1789760791.974694:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80666624 chunk: [80666624, 80670720) 80670720, 4096 00020000:00200000:0.0:1789760791.974699:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.974724:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19694, 19694] 00000008:00010000:0.0:1789760791.974744:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.974750:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.974761:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80666624, 80670720) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.974940:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80666624=0x4cee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.975007:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.975010:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80670720 00000020:00010000:0.0:1789760791.975020:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.975023:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.975027:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.975030:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.975154:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.975166:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.975175:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.975194:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760791.975302:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99c00 - object 0x280000401:2 offset 80666624<>80670720 00000100:00100000:1.0:1789760791.975440:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503910400:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760791.988706:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987001 gen 2 00000008:00010000:1.0:1789760791.988775:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760791.988784:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.988798:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.988823:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760791.988857:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503910400:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760791.988883:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19694, 19694] 00000080:00200000:0.0:1789760791.988899:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.988906:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.988923:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80670720 i_blocks 157584 00000020:00200000:0.0:1789760791.988927:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.988937:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.988978:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80670720, count: 4096 00000080:00200000:0.0:1789760791.988986:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80670720, bytes: 4096 00000080:00200000:0.0:1789760791.988996:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.989006:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760791.989008:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80670720, 80674816) 0 0 00000080:00200000:0.0:1789760791.989012:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.989029:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19695, 19695] 00020000:00200000:0.0:1789760791.989070:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80670720 chunk: [80670720, 80674816) 80674816, 4096 00020000:00200000:0.0:1789760791.989076:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.989100:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19695, 19695] 00000008:00010000:0.0:1789760791.989119:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.989126:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760791.989138:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80670720, 80674816) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.989154:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80670720=0x4cef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.989219:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.989222:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80674816 00000020:00010000:0.0:1789760791.989231:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.989234:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.989238:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.989242:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.989244:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.989252:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.989259:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.989278:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760791.990965:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a300 - object 0x280000401:2 offset 80670720<>80674816 00000100:00100000:1.0:1789760791.991670:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503910528:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.013985:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987002 gen 2 00000008:00010000:1.0:1789760792.014064:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760792.014073:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.014084:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.014164:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760792.014198:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503910528:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.020342:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19695, 19695] 00000080:00200000:0.0:1789760792.020485:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.020493:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.020514:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80674816 i_blocks 157592 00000020:00200000:0.0:1789760792.020520:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.020538:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.021218:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80674816, count: 4096 00000080:00200000:0.0:1789760792.021231:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80674816, bytes: 4096 00000080:00200000:0.0:1789760792.021246:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.021256:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.021258:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80674816, 80678912) 0 0 00000080:00200000:0.0:1789760792.021262:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.021281:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19696, 19696] 00020000:00200000:0.0:1789760792.021291:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80674816 chunk: [80674816, 80678912) 80678912, 4096 00020000:00200000:0.0:1789760792.021297:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.021319:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19696, 19696] 00000008:00010000:0.0:1789760792.021339:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.021345:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.021357:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80674816, 80678912) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.021377:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80674816=0x4cf0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.021449:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.021451:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80678912 00000020:00010000:0.0:1789760792.021459:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.021462:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.021466:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.021470:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.021473:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.021482:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.021490:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.021512:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760792.022658:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99500 - object 0x280000401:2 offset 80674816<>80678912 00000100:00100000:1.0:1789760792.023924:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760792.023965:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503910656:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.045227:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987003 gen 2 00000008:00010000:1.0:1789760792.045283:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760792.045293:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.045306:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.045432:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760792.045469:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503910656:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.045909:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19696, 19696] 00000080:00200000:0.0:1789760792.045927:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.045933:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.045950:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80678912 i_blocks 157600 00000020:00200000:0.0:1789760792.045954:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.045964:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.046002:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80678912, count: 4096 00000080:00200000:0.0:1789760792.046009:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80678912, bytes: 4096 00000080:00200000:0.0:1789760792.046025:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.050305:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.050322:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80678912, 80683008) 0 0 00000080:00200000:0.0:1789760792.050332:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.050357:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19697, 19697] 00020000:00200000:0.0:1789760792.050370:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80678912 chunk: [80678912, 80683008) 80683008, 4096 00020000:00200000:0.0:1789760792.050558:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.050596:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19697, 19697] 00000008:00010000:0.0:1789760792.050624:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.050708:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.050725:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80678912, 80683008) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.050742:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80678912=0x4cf1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.050904:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.050908:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80683008 00000020:00010000:0.0:1789760792.050918:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.050921:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.050925:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.050929:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.050933:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.050948:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.050956:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.050989:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760792.051129:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99500 - object 0x280000401:2 offset 80678912<>80683008 00000100:00100000:1.0:1789760792.051491:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503910784:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.063242:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987004 gen 2 00000008:00010000:1.0:1789760792.063296:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760792.063305:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.063329:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.063354:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760792.063394:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503910784:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.063957:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19697, 19697] 00000080:00200000:0.0:1789760792.063976:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.063983:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.064000:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80683008 i_blocks 157608 00000020:00200000:0.0:1789760792.064004:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.064014:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.064149:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80683008, count: 4096 00000080:00200000:0.0:1789760792.064157:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80683008, bytes: 4096 00000080:00200000:0.0:1789760792.064169:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.064178:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.064181:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80683008, 80687104) 0 0 00000080:00200000:0.0:1789760792.064185:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.064201:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19698, 19698] 00020000:00200000:0.0:1789760792.064210:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80683008 chunk: [80683008, 80687104) 80687104, 4096 00020000:00200000:0.0:1789760792.064216:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.064239:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19698, 19698] 00000008:00010000:0.0:1789760792.064259:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.064265:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.064279:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80683008, 80687104) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.064295:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80683008=0x4cf2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.064365:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.064368:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80687104 00000020:00010000:0.0:1789760792.064377:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.064380:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.064384:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.064388:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.064391:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.064401:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.064408:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.064423:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760792.065070:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a300 - object 0x280000401:2 offset 80683008<>80687104 00000100:00100000:1.0:1789760792.065269:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760792.065295:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503910912:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.076488:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987005 gen 2 00000008:00010000:1.0:1789760792.076538:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760792.076547:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.076559:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.076600:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760792.076632:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503910912:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.076647:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19698, 19698] 00000080:00200000:0.0:1789760792.076664:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.076670:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.076686:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80687104 i_blocks 157616 00000020:00200000:0.0:1789760792.076690:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.076699:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.076736:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80687104, count: 4096 00000080:00200000:0.0:1789760792.076744:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80687104, bytes: 4096 00000080:00200000:0.0:1789760792.076758:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.076767:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.076770:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80687104, 80691200) 0 0 00000080:00200000:0.0:1789760792.076774:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.076794:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19699, 19699] 00020000:00200000:0.0:1789760792.076804:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80687104 chunk: [80687104, 80691200) 80691200, 4096 00020000:00200000:0.0:1789760792.076809:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.076832:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19699, 19699] 00000008:00010000:0.0:1789760792.076853:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.076859:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.076881:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80687104, 80691200) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.076897:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80687104=0x4cf3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.076958:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.076961:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80691200 00000020:00010000:0.0:1789760792.076970:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.076974:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.076978:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.076981:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.076984:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.076992:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.077000:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.077016:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760792.077142:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99c00 - object 0x280000401:2 offset 80687104<>80691200 00000100:00100000:1.0:1789760792.078293:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503911040:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.089954:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987006 gen 2 00000008:00010000:1.0:1789760792.090006:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760792.090014:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.090027:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.090096:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760792.090128:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503911040:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.094467:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19699, 19699] 00000080:00200000:0.0:1789760792.094496:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.094503:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.094522:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80691200 i_blocks 157624 00000020:00200000:0.0:1789760792.094528:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.094546:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.094580:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80691200, count: 4096 00000080:00200000:0.0:1789760792.094599:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80691200, bytes: 4096 00000080:00200000:0.0:1789760792.094616:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.094626:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.094628:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80691200, 80695296) 0 0 00000080:00200000:0.0:1789760792.094632:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.094648:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19700, 19700] 00020000:00200000:0.0:1789760792.094657:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80691200 chunk: [80691200, 80695296) 80695296, 4096 00020000:00200000:0.0:1789760792.094662:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.094685:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19700, 19700] 00000008:00010000:0.0:1789760792.094709:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.094716:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.094729:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80691200, 80695296) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.094746:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80691200=0x4cf4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.094817:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.094820:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80695296 00000020:00010000:0.0:1789760792.094827:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.094831:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.094835:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.094838:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.094842:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.094851:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.094858:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.094882:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760792.095031:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98380 - object 0x280000401:2 offset 80691200<>80695296 00000100:00100000:1.0:1789760792.095127:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503911168:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.119955:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987007 gen 2 00000008:00010000:1.0:1789760792.120005:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760792.120014:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.120027:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.120089:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760792.120123:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503911168:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760792.120515:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19700, 19700] 00000080:00200000:3.0:1789760792.120530:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.120536:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760792.120552:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80695296 i_blocks 157632 00000020:00200000:3.0:1789760792.120557:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760792.120566:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760792.120610:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80695296, count: 4096 00000080:00200000:3.0:1789760792.120617:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80695296, bytes: 4096 00000080:00200000:3.0:1789760792.120628:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760792.120637:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760792.120640:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80695296, 80699392) 0 0 00000080:00200000:3.0:1789760792.120644:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760792.120674:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19701, 19701] 00020000:00200000:3.0:1789760792.120683:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80695296 chunk: [80695296, 80699392) 80699392, 4096 00020000:00200000:3.0:1789760792.120688:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760792.120708:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19701, 19701] 00000008:00010000:3.0:1789760792.120727:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760792.120733:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760792.120744:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80695296, 80699392) append 0 append lockpos 0 00000080:00200000:3.0:1789760792.120757:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80695296=0x4cf5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760792.120818:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760792.120821:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80699392 00000020:00010000:3.0:1789760792.120829:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760792.120832:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.120836:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.120839:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760792.120841:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760792.120849:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760792.120857:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760792.120874:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1f00 00000008:00100000:3.0:1789760792.120980:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f480 - object 0x280000401:2 offset 80695296<>80699392 00000100:00100000:3.0:1789760792.121021:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760792.121068:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503911296:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760792.141514:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19701, 19701] 00000080:00200000:1.0:1789760792.141532:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.141538:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.141555:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80699392 i_blocks 157640 00000020:00200000:1.0:1789760792.141559:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.141570:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.141613:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80699392, count: 4096 00000080:00200000:1.0:1789760792.141620:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80699392, bytes: 4096 00000080:00200000:1.0:1789760792.141712:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.141724:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760792.141726:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80699392, 80703488) 0 0 00000080:00200000:1.0:1789760792.141731:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.141750:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19702, 19702] 00020000:00200000:1.0:1789760792.141758:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80699392 chunk: [80699392, 80703488) 80703488, 4096 00020000:00200000:1.0:1789760792.141764:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.141786:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19702, 19702] 00000008:00010000:1.0:1789760792.141804:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.141809:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760792.141821:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80699392, 80703488) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.141835:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80699392=0x4cf6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.141893:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.141896:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80703488 00000020:00010000:1.0:1789760792.141904:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.141908:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.141911:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.141915:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.141917:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.141926:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.141932:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.141948:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760792.142074:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 80699392<>80703488 00000100:00100000:0.0:1789760792.142161:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503911424:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760792.153399:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987009 gen 2 00000008:00010000:0.0:1789760792.153453:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:0.0:1789760792.153462:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.153474:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.153501:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000100:00100000:0.0:1789760792.153533:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503911424:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760792.153772:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19702, 19702] 00000080:00200000:1.0:1789760792.153790:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.153797:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.153815:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80703488 i_blocks 157648 00000020:00200000:1.0:1789760792.153819:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.153829:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.153868:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80703488, count: 4096 00000080:00200000:1.0:1789760792.153875:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80703488, bytes: 4096 00000080:00200000:1.0:1789760792.153886:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.153895:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760792.153897:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80703488, 80707584) 0 0 00000080:00200000:1.0:1789760792.153901:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.153919:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19703, 19703] 00020000:00200000:1.0:1789760792.153928:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80703488 chunk: [80703488, 80707584) 80707584, 4096 00020000:00200000:1.0:1789760792.153933:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.153953:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19703, 19703] 00000008:00010000:1.0:1789760792.153973:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.153980:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760792.153992:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80703488, 80707584) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.154007:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80703488=0x4cf7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.154113:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.154116:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80707584 00000020:00010000:1.0:1789760792.154124:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.154127:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.154131:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.154134:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.154137:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.154574:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.154583:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.154615:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760792.154738:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 80703488<>80707584 00000100:00100000:0.0:1789760792.155362:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760792.155388:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503911552:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760792.181865:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987010 gen 2 00000008:00010000:0.0:1789760792.181926:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58000 00000020:00200000:0.0:1789760792.181934:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.181946:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.181975:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58000 00000100:00100000:0.0:1789760792.182007:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503911552:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760792.182255:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19703, 19703] 00000080:00200000:1.0:1789760792.182272:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.182278:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.182295:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80707584 i_blocks 157656 00000020:00200000:1.0:1789760792.182299:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.182309:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.182345:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80707584, count: 4096 00000080:00200000:1.0:1789760792.182353:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80707584, bytes: 4096 00000080:00200000:1.0:1789760792.182366:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.182375:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760792.182377:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80707584, 80711680) 0 0 00000080:00200000:1.0:1789760792.182382:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.182399:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19704, 19704] 00020000:00200000:1.0:1789760792.182408:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80707584 chunk: [80707584, 80711680) 80711680, 4096 00020000:00200000:1.0:1789760792.182413:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.182434:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19704, 19704] 00000008:00010000:1.0:1789760792.182454:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.182460:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760792.182471:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80707584, 80711680) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.182486:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80707584=0x4cf8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.182661:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.182664:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80711680 00000020:00010000:1.0:1789760792.182674:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.182678:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.182681:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.182684:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.182687:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.182696:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.182703:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.182721:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760792.182836:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 80707584<>80711680 00000100:00100000:0.0:1789760792.183187:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503911680:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760792.198256:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987011 gen 2 00000008:00010000:0.0:1789760792.198319:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:0.0:1789760792.198328:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.198341:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.198354:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:0.0:1789760792.198387:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503911680:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760792.198656:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19704, 19704] 00000080:00200000:1.0:1789760792.198672:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.198678:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.198695:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80711680 i_blocks 157664 00000020:00200000:1.0:1789760792.198700:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.198710:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.198760:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80711680, count: 4096 00000080:00200000:1.0:1789760792.198768:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80711680, bytes: 4096 00000080:00200000:1.0:1789760792.198780:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.198791:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760792.198794:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80711680, 80715776) 0 0 00000080:00200000:1.0:1789760792.198798:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.198817:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19705, 19705] 00020000:00200000:1.0:1789760792.198825:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80711680 chunk: [80711680, 80715776) 80715776, 4096 00020000:00200000:1.0:1789760792.198830:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.198851:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19705, 19705] 00000008:00010000:1.0:1789760792.198870:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.198876:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760792.198888:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80711680, 80715776) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.199030:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80711680=0x4cf9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.199122:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.199125:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80715776 00000020:00010000:1.0:1789760792.199133:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.199137:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.199140:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.199144:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.199148:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.199157:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.199164:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.199182:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760792.199303:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 80711680<>80715776 00000100:00100000:0.0:1789760792.199410:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503911808:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760792.219392:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987012 gen 2 00000008:00010000:0.0:1789760792.219450:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d586c0 00000020:00200000:0.0:1789760792.219459:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.219470:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.219565:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d586c0 00000100:00100000:0.0:1789760792.219607:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503911808:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760792.227204:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503911936:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.242827:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19706, 19706] 00000080:00200000:0.0:1789760792.242850:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.242857:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.242874:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80719872 i_blocks 157680 00000020:00200000:0.0:1789760792.242879:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.242892:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.243152:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80719872, count: 4096 00000080:00200000:0.0:1789760792.243162:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80719872, bytes: 4096 00000080:00200000:0.0:1789760792.243176:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.243185:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.243188:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80719872, 80723968) 0 0 00000080:00200000:0.0:1789760792.243192:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.243211:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19707, 19707] 00020000:00200000:0.0:1789760792.243221:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80719872 chunk: [80719872, 80723968) 80723968, 4096 00020000:00200000:0.0:1789760792.243227:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.243248:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19707, 19707] 00000008:00010000:0.0:1789760792.243268:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.243275:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.243286:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80719872, 80723968) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.243303:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80719872=0x4cfb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.243376:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.243379:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80723968 00000020:00010000:0.0:1789760792.243388:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.243392:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.243395:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.243399:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.243402:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.243411:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.243418:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.243448:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760792.243561:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9ad80 - object 0x280000401:2 offset 80719872<>80723968 00000100:00100000:1.0:1789760792.243655:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503912064:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.259361:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987014 gen 2 00000008:00010000:1.0:1789760792.259417:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760792.259428:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.259443:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.259478:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760792.259522:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503912064:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760792.260154:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503912192:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760792.264363:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503912320:192.168.203.158@tcp:400:kworker.0 00000080:00200000:3.0:1789760792.275574:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19708, 19708] 00000080:00200000:3.0:1789760792.275707:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.275714:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760792.275731:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80728064 i_blocks 157696 00000020:00200000:3.0:1789760792.275735:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760792.275746:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760792.275784:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80728064, count: 4096 00000080:00200000:3.0:1789760792.275792:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80728064, bytes: 4096 00000080:00200000:3.0:1789760792.275803:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760792.275812:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760792.275814:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80728064, 80732160) 0 0 00000080:00200000:3.0:1789760792.275821:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760792.275838:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19709, 19709] 00020000:00200000:3.0:1789760792.275847:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80728064 chunk: [80728064, 80732160) 80732160, 4096 00020000:00200000:3.0:1789760792.275852:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760792.275947:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19709, 19709] 00000008:00010000:3.0:1789760792.275970:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760792.275977:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760792.275989:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80728064, 80732160) append 0 append lockpos 0 00000080:00200000:3.0:1789760792.276004:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80728064=0x4cfd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760792.276107:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760792.276111:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80732160 00000020:00010000:3.0:1789760792.276120:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760792.276123:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.276126:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.276130:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760792.276133:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760792.276140:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760792.276148:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760792.276166:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1540 00000008:00100000:3.0:1789760792.276304:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ed80 - object 0x280000401:2 offset 80728064<>80732160 00000100:00100000:3.0:1789760792.276350:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760792.276370:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503912960:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.287927:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19709, 19709] 00000080:00200000:0.0:1789760792.287944:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.287951:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.287967:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80732160 i_blocks 157704 00000020:00200000:0.0:1789760792.287972:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.287984:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.288113:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80732160, count: 4096 00000080:00200000:0.0:1789760792.288329:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80732160, bytes: 4096 00000080:00200000:0.0:1789760792.288345:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.288354:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.288356:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80732160, 80736256) 0 0 00000080:00200000:0.0:1789760792.288362:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.288379:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19710, 19710] 00020000:00200000:0.0:1789760792.288389:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80732160 chunk: [80732160, 80736256) 80736256, 4096 00020000:00200000:0.0:1789760792.288394:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.288415:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19710, 19710] 00000008:00010000:0.0:1789760792.288435:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.288442:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.288453:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80732160, 80736256) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.288468:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80732160=0x4cfe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.288533:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.288537:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80736256 00000020:00010000:0.0:1789760792.288545:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.288548:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.288552:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.288555:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.288558:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.288567:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.288574:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.288603:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760792.288710:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b100 - object 0x280000401:2 offset 80732160<>80736256 00000100:00100000:1.0:1789760792.288804:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503913088:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.305755:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987017 gen 2 00000008:00010000:1.0:1789760792.305806:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760792.305814:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.305826:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.305909:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760792.305942:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503913088:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.306809:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80740352 00000020:00010000:2.0:1789760792.306819:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.306823:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.306826:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.306829:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.306833:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.306841:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.306849:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.306870:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.306981:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80736256<>80740352 00000100:00100000:3.0:1789760792.307072:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503913216:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.319933:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987018 gen 2 00000008:00010000:2.0:1789760792.319984:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.319992:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.320004:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.320376:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.320408:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503913216:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.321910:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19711, 19711] 00000080:00200000:0.0:1789760792.321934:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.321941:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.321959:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80740352 i_blocks 157720 00000020:00200000:0.0:1789760792.321965:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.321979:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.322010:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80740352, count: 4096 00000080:00200000:0.0:1789760792.322291:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80740352, bytes: 4096 00000080:00200000:0.0:1789760792.322376:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.322386:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.322389:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80740352, 80744448) 0 0 00000080:00200000:0.0:1789760792.322394:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.322413:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19712, 19712] 00020000:00200000:0.0:1789760792.322422:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80740352 chunk: [80740352, 80744448) 80744448, 4096 00020000:00200000:0.0:1789760792.322426:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.322448:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19712, 19712] 00000008:00010000:0.0:1789760792.322467:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.322474:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.322485:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80740352, 80744448) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.322500:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80740352=0x4d00000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.322576:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.322579:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80744448 00000020:00010000:0.0:1789760792.322596:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.322599:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.322602:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.322605:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.322608:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.322617:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.322691:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.322721:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760792.322838:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9aa00 - object 0x280000401:2 offset 80740352<>80744448 00000100:00100000:1.0:1789760792.322969:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503913344:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.341559:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987019 gen 2 00000008:00010000:1.0:1789760792.341685:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760792.341695:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.341764:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.341788:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760792.341819:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503913344:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.341930:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19712, 19712] 00000080:00200000:0.0:1789760792.341947:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.341953:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.341969:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80744448 i_blocks 157728 00000020:00200000:0.0:1789760792.341973:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.341983:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.342102:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80744448, count: 4096 00000080:00200000:0.0:1789760792.342109:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80744448, bytes: 4096 00000080:00200000:0.0:1789760792.342121:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.342129:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.342132:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80744448, 80748544) 0 0 00000080:00200000:0.0:1789760792.342136:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.342291:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19713, 19713] 00020000:00200000:0.0:1789760792.342300:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80744448 chunk: [80744448, 80748544) 80748544, 4096 00020000:00200000:0.0:1789760792.342305:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.342326:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19713, 19713] 00000008:00010000:0.0:1789760792.342345:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.342352:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.342363:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80744448, 80748544) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.342378:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80744448=0x4d01000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.342440:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.342443:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80748544 00000020:00010000:0.0:1789760792.342452:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.342455:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.342458:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.342462:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.342465:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.342474:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.342481:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.342500:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760792.342620:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b800 - object 0x280000401:2 offset 80744448<>80748544 00000100:00100000:1.0:1789760792.342811:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503913472:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.354024:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987020 gen 2 00000008:00010000:1.0:1789760792.354113:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760792.354122:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.354135:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.354158:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760792.354189:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503913472:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.354408:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19713, 19713] 00000080:00200000:0.0:1789760792.354425:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.354432:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.354450:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80748544 i_blocks 157736 00000020:00200000:0.0:1789760792.354455:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.354466:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.354502:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80748544, count: 4096 00000080:00200000:0.0:1789760792.354509:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80748544, bytes: 4096 00000080:00200000:0.0:1789760792.354520:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.354529:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.354531:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80748544, 80752640) 0 0 00000080:00200000:0.0:1789760792.354536:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.354553:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19714, 19714] 00020000:00200000:0.0:1789760792.354562:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80748544 chunk: [80748544, 80752640) 80752640, 4096 00020000:00200000:0.0:1789760792.354568:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.354599:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19714, 19714] 00000008:00010000:0.0:1789760792.354618:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.354624:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.354636:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80748544, 80752640) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.354650:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80748544=0x4d02000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.354774:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.354778:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80752640 00000020:00010000:0.0:1789760792.354787:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.354790:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.354794:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.354797:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.354800:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.354809:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.354816:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.354833:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760792.354938:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b480 - object 0x280000401:2 offset 80748544<>80752640 00000100:00100000:1.0:1789760792.355525:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503913600:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.375523:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987021 gen 2 00000008:00010000:1.0:1789760792.375578:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760792.375597:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.375610:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.375635:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760792.375668:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503913600:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.375881:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19714, 19714] 00000080:00200000:0.0:1789760792.375897:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.375904:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.375922:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80752640 i_blocks 157744 00000020:00200000:0.0:1789760792.375926:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.375936:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.375972:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80752640, count: 4096 00000080:00200000:0.0:1789760792.375980:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80752640, bytes: 4096 00000080:00200000:0.0:1789760792.375991:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.376000:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.376002:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80752640, 80756736) 0 0 00000080:00200000:0.0:1789760792.376007:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.376022:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19715, 19715] 00020000:00200000:0.0:1789760792.376030:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80752640 chunk: [80752640, 80756736) 80756736, 4096 00020000:00200000:0.0:1789760792.376058:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.376079:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19715, 19715] 00000008:00010000:0.0:1789760792.376098:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.376105:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.376117:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80752640, 80756736) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.376132:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80752640=0x4d03000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.376192:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.376195:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80756736 00000020:00010000:0.0:1789760792.376204:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.376207:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.376211:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.376215:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.376218:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.376226:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.376234:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.376251:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760792.376372:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a680 - object 0x280000401:2 offset 80752640<>80756736 00000100:00100000:1.0:1789760792.376709:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760792.376733:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503913728:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.386766:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987022 gen 2 00000008:00010000:1.0:1789760792.386821:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760792.386830:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.386843:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.386867:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760792.386898:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503913728:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.387013:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19715, 19715] 00000080:00200000:0.0:1789760792.387030:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.387075:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.387093:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80756736 i_blocks 157752 00000020:00200000:0.0:1789760792.387097:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.387107:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.387144:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80756736, count: 4096 00000080:00200000:0.0:1789760792.387151:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80756736, bytes: 4096 00000080:00200000:0.0:1789760792.387161:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.387169:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.387171:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80756736, 80760832) 0 0 00000080:00200000:0.0:1789760792.387175:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.387193:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19716, 19716] 00020000:00200000:0.0:1789760792.387202:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80756736 chunk: [80756736, 80760832) 80760832, 4096 00020000:00200000:0.0:1789760792.387207:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.387228:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19716, 19716] 00000008:00010000:0.0:1789760792.387246:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.387252:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.387264:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80756736, 80760832) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.387278:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80756736=0x4d04000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.387338:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.387341:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80760832 00000020:00010000:0.0:1789760792.387349:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.387353:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.387356:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.387360:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.387363:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.387371:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.387379:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.387395:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760792.387503:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98000 - object 0x280000401:2 offset 80756736<>80760832 00000100:00100000:1.0:1789760792.387611:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503913856:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.401945:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987023 gen 2 00000008:00010000:1.0:1789760792.401998:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760792.402006:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.402020:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.402070:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760792.402102:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503913856:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.402208:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19716, 19716] 00000080:00200000:0.0:1789760792.402225:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.402232:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.402248:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80760832 i_blocks 157760 00000020:00200000:0.0:1789760792.402252:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.402262:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.402298:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80760832, count: 4096 00000080:00200000:0.0:1789760792.402305:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80760832, bytes: 4096 00000080:00200000:0.0:1789760792.402374:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.402384:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.402386:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80760832, 80764928) 0 0 00000080:00200000:0.0:1789760792.402391:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.402409:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19717, 19717] 00020000:00200000:0.0:1789760792.402550:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80760832 chunk: [80760832, 80764928) 80764928, 4096 00020000:00200000:0.0:1789760792.402555:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.402576:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19717, 19717] 00000008:00010000:0.0:1789760792.402605:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.402611:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.402680:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80760832, 80764928) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.402694:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80760832=0x4d05000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.402753:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.402756:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80764928 00000020:00010000:0.0:1789760792.402764:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.402767:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.402771:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.402774:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.402777:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.402785:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.402791:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.402809:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760792.402912:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98a80 - object 0x280000401:2 offset 80760832<>80764928 00000100:00100000:1.0:1789760792.403016:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760792.403130:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503913984:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.413950:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987024 gen 2 00000008:00010000:1.0:1789760792.414007:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760792.414016:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.414029:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.414093:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760792.414124:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503913984:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.414277:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19717, 19717] 00000080:00200000:0.0:1789760792.414295:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.414301:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.414329:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80764928 i_blocks 157768 00000020:00200000:0.0:1789760792.414333:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.414344:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.414381:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80764928, count: 4096 00000080:00200000:0.0:1789760792.414388:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80764928, bytes: 4096 00000080:00200000:0.0:1789760792.414398:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.414407:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.414409:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80764928, 80769024) 0 0 00000080:00200000:0.0:1789760792.414414:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.414430:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19718, 19718] 00020000:00200000:0.0:1789760792.414438:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80764928 chunk: [80764928, 80769024) 80769024, 4096 00020000:00200000:0.0:1789760792.414443:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.414463:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19718, 19718] 00000008:00010000:0.0:1789760792.414548:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.414556:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.414569:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80764928, 80769024) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.414593:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80764928=0x4d06000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.414653:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.414656:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80769024 00000020:00010000:0.0:1789760792.414665:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.414668:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.414671:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.414675:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.414678:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.414686:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.414694:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.414712:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760792.414819:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 80764928<>80769024 00000100:00100000:1.0:1789760792.414881:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503914112:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.426151:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987025 gen 2 00000008:00010000:1.0:1789760792.426202:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760792.426210:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.426223:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.426249:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760792.426283:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503914112:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.426537:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19718, 19718] 00000080:00200000:2.0:1789760792.426555:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.426561:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.426577:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80769024 i_blocks 157776 00000020:00200000:2.0:1789760792.426581:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.426601:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.426648:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80769024, count: 4096 00000080:00200000:2.0:1789760792.426656:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80769024, bytes: 4096 00000080:00200000:2.0:1789760792.426672:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.426682:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.426684:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80769024, 80773120) 0 0 00000080:00200000:2.0:1789760792.426688:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.426706:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19719, 19719] 00020000:00200000:2.0:1789760792.426716:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80769024 chunk: [80769024, 80773120) 80773120, 4096 00020000:00200000:2.0:1789760792.426721:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.426743:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19719, 19719] 00000008:00010000:2.0:1789760792.426762:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.426769:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.426780:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80769024, 80773120) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.426795:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80769024=0x4d07000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.426858:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.426862:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80773120 00000020:00010000:2.0:1789760792.426871:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.426874:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.426877:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.426880:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.426883:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.426891:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.426898:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.426917:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.427023:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80769024<>80773120 00000100:00100000:2.0:1789760792.427118:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760792.427135:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503914240:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.436293:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987026 gen 2 00000008:00010000:2.0:1789760792.436352:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.436362:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.436374:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.436400:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.436433:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503914240:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.436613:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19719, 19719] 00000080:00200000:0.0:1789760792.436637:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.436643:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.436659:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80773120 i_blocks 157784 00000020:00200000:0.0:1789760792.436663:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.436674:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.436710:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80773120, count: 4096 00000080:00200000:0.0:1789760792.436717:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80773120, bytes: 4096 00000080:00200000:0.0:1789760792.436728:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.436737:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.436740:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80773120, 80777216) 0 0 00000080:00200000:0.0:1789760792.436745:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.436760:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19720, 19720] 00020000:00200000:0.0:1789760792.436770:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80773120 chunk: [80773120, 80777216) 80777216, 4096 00020000:00200000:0.0:1789760792.436774:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.436795:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19720, 19720] 00000008:00010000:0.0:1789760792.436812:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.436817:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.436827:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80773120, 80777216) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.436841:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80773120=0x4d08000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.436895:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.436898:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80777216 00000020:00010000:0.0:1789760792.436905:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.436908:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.436911:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.436914:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.436917:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.436924:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.436929:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.436943:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760792.437064:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99880 - object 0x280000401:2 offset 80773120<>80777216 00000100:00100000:1.0:1789760792.437240:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503914368:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.445991:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987027 gen 2 00000008:00010000:1.0:1789760792.446076:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760792.446089:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.446102:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.446125:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760792.446159:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503914368:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.446278:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19720, 19720] 00000080:00200000:0.0:1789760792.446295:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.446301:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.446317:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80777216 i_blocks 157792 00000020:00200000:0.0:1789760792.446321:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.446332:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.446368:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80777216, count: 4096 00000080:00200000:0.0:1789760792.446375:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80777216, bytes: 4096 00000080:00200000:0.0:1789760792.446385:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.446394:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.446396:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80777216, 80781312) 0 0 00000080:00200000:0.0:1789760792.446401:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.446419:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19721, 19721] 00020000:00200000:0.0:1789760792.446429:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80777216 chunk: [80777216, 80781312) 80781312, 4096 00020000:00200000:0.0:1789760792.446434:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.446454:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19721, 19721] 00000008:00010000:0.0:1789760792.446473:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.446479:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.446490:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80777216, 80781312) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.446505:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80777216=0x4d09000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.446562:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.446566:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80781312 00000020:00010000:0.0:1789760792.446575:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.446579:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.446582:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.446595:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.446598:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.446607:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.446615:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.446834:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760792.446945:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99f80 - object 0x280000401:2 offset 80777216<>80781312 00000100:00100000:1.0:1789760792.447015:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503914496:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.455008:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987028 gen 2 00000008:00010000:1.0:1789760792.455087:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760792.455096:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.455110:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.455137:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760792.455169:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503914496:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.455338:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19721, 19721] 00000080:00200000:2.0:1789760792.455354:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.455361:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.455378:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80781312 i_blocks 157800 00000020:00200000:2.0:1789760792.455382:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.455392:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.455427:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80781312, count: 4096 00000080:00200000:2.0:1789760792.455435:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80781312, bytes: 4096 00000080:00200000:2.0:1789760792.455447:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.455456:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.455459:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80781312, 80785408) 0 0 00000080:00200000:2.0:1789760792.455464:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.455479:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19722, 19722] 00020000:00200000:2.0:1789760792.455488:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80781312 chunk: [80781312, 80785408) 80785408, 4096 00020000:00200000:2.0:1789760792.455504:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.455524:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19722, 19722] 00000008:00010000:2.0:1789760792.455541:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.455548:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.455560:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80781312, 80785408) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.455574:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80781312=0x4d0a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.455840:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.455843:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80785408 00000020:00010000:2.0:1789760792.455854:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.455858:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.455862:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.455866:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.455869:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.455878:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.455887:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.455905:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.456013:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80781312<>80785408 00000100:00100000:3.0:1789760792.456284:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760792.456414:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503914624:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.465988:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987029 gen 2 00000008:00010000:2.0:1789760792.466070:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.466079:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.466090:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.466119:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.466150:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503914624:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760792.466278:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19722, 19722] 00000080:00200000:1.0:1789760792.466297:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.466304:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.466329:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80785408 i_blocks 157808 00000020:00200000:1.0:1789760792.466334:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.466345:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.466384:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80785408, count: 4096 00000080:00200000:1.0:1789760792.466391:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80785408, bytes: 4096 00000080:00200000:1.0:1789760792.466405:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.466414:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760792.466416:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80785408, 80789504) 0 0 00000080:00200000:1.0:1789760792.466421:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.466439:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19723, 19723] 00020000:00200000:1.0:1789760792.466447:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80785408 chunk: [80785408, 80789504) 80789504, 4096 00020000:00200000:1.0:1789760792.466452:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.466472:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19723, 19723] 00000008:00010000:1.0:1789760792.466491:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.466497:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760792.466508:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80785408, 80789504) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.466522:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80785408=0x4d0b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.466594:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.466598:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80789504 00000020:00010000:1.0:1789760792.466607:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.466611:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.466614:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.466617:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.466630:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.466639:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.466646:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.466662:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760792.466781:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 80785408<>80789504 00000100:00100000:1.0:1789760792.466830:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760792.466848:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503914752:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.476117:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987030 gen 2 00000008:00010000:1.0:1789760792.476174:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58a80 00000020:00200000:1.0:1789760792.476182:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.476195:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.476225:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58a80 00000100:00100000:1.0:1789760792.476261:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503914752:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.476444:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19723, 19723] 00000080:00200000:2.0:1789760792.476462:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.476469:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.476486:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80789504 i_blocks 157816 00000020:00200000:2.0:1789760792.476490:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.476502:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.476537:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80789504, count: 4096 00000080:00200000:2.0:1789760792.476544:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80789504, bytes: 4096 00000080:00200000:2.0:1789760792.476555:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.476565:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.476567:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80789504, 80793600) 0 0 00000080:00200000:2.0:1789760792.476573:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.476600:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19724, 19724] 00020000:00200000:2.0:1789760792.476610:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80789504 chunk: [80789504, 80793600) 80793600, 4096 00020000:00200000:2.0:1789760792.476615:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.476707:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19724, 19724] 00000008:00010000:2.0:1789760792.476731:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.476737:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.476751:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80789504, 80793600) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.476767:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80789504=0x4d0c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.476829:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.476832:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80793600 00000020:00010000:2.0:1789760792.476841:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.476844:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.476847:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.476851:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.476854:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.476863:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.476870:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.476889:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.476994:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80789504<>80793600 00000100:00100000:3.0:1789760792.477520:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503914880:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.495539:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987031 gen 2 00000008:00010000:2.0:1789760792.495601:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.495609:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.495620:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.495646:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.495678:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503914880:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760792.495812:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19724, 19724] 00000080:00200000:1.0:1789760792.495829:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.495836:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.495852:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80793600 i_blocks 157824 00000020:00200000:1.0:1789760792.495855:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.495866:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.495902:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80793600, count: 4096 00000080:00200000:1.0:1789760792.495909:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80793600, bytes: 4096 00000080:00200000:1.0:1789760792.495918:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.495927:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760792.495929:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80793600, 80797696) 0 0 00000080:00200000:1.0:1789760792.495934:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.495951:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19725, 19725] 00020000:00200000:1.0:1789760792.495960:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80793600 chunk: [80793600, 80797696) 80797696, 4096 00020000:00200000:1.0:1789760792.495965:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.495984:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19725, 19725] 00000008:00010000:1.0:1789760792.496002:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.496009:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760792.496020:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80793600, 80797696) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.496066:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80793600=0x4d0d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.496124:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.496127:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80797696 00000020:00010000:1.0:1789760792.496136:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.496140:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.496143:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.496146:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.496149:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.496157:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.496164:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.496182:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760792.496281:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 80793600<>80797696 00000100:00100000:0.0:1789760792.496357:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503915008:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760792.508907:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987032 gen 2 00000008:00010000:0.0:1789760792.508962:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58a80 00000020:00200000:0.0:1789760792.508971:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.508982:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.509137:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58a80 00000100:00100000:0.0:1789760792.509167:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503915008:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760792.509380:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19725, 19725] 00000080:00200000:1.0:1789760792.509444:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.509451:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.509468:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80797696 i_blocks 157832 00000020:00200000:1.0:1789760792.509472:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.509483:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.509529:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80797696, count: 4096 00000080:00200000:1.0:1789760792.509536:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80797696, bytes: 4096 00000080:00200000:1.0:1789760792.509547:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.509556:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760792.509559:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80797696, 80801792) 0 0 00000080:00200000:1.0:1789760792.509564:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.509590:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19726, 19726] 00020000:00200000:1.0:1789760792.509599:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80797696 chunk: [80797696, 80801792) 80801792, 4096 00020000:00200000:1.0:1789760792.509605:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.509782:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19726, 19726] 00000008:00010000:1.0:1789760792.509802:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.509808:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760792.509820:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80797696, 80801792) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.509834:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80797696=0x4d0e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.509895:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.509898:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80801792 00000020:00010000:1.0:1789760792.509907:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.509911:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.509913:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.509917:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.509920:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.509927:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.509934:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.509954:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760792.510088:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 80797696<>80801792 00000100:00100000:0.0:1789760792.510162:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503915136:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760792.524646:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987033 gen 2 00000008:00010000:0.0:1789760792.524702:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58540 00000020:00200000:0.0:1789760792.524710:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.524721:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.524752:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58540 00000100:00100000:0.0:1789760792.524787:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503915136:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.525023:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19726, 19726] 00000080:00200000:2.0:1789760792.525088:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.525095:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.525111:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80801792 i_blocks 157840 00000020:00200000:2.0:1789760792.525115:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.525125:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.525160:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80801792, count: 4096 00000080:00200000:2.0:1789760792.525167:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80801792, bytes: 4096 00000080:00200000:2.0:1789760792.525180:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.525189:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.525191:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80801792, 80805888) 0 0 00000080:00200000:2.0:1789760792.525196:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.525213:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19727, 19727] 00020000:00200000:2.0:1789760792.525221:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80801792 chunk: [80801792, 80805888) 80805888, 4096 00020000:00200000:2.0:1789760792.525226:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.525246:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19727, 19727] 00000008:00010000:2.0:1789760792.525264:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.525270:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.525282:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80801792, 80805888) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.525297:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80801792=0x4d0f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.525869:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.525874:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80805888 00000020:00010000:2.0:1789760792.525885:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.525889:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.525892:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.525895:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.525959:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.525971:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.525981:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.526000:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.526131:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80801792<>80805888 00000100:00100000:3.0:1789760792.526204:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503915264:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.538102:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987034 gen 2 00000008:00010000:2.0:1789760792.538160:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.538170:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.538181:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.538210:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.538241:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503915264:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.538364:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19727, 19727] 00000080:00200000:0.0:1789760792.538382:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.538388:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.538404:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80805888 i_blocks 157848 00000020:00200000:0.0:1789760792.538408:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.538419:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.538454:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80805888, count: 4096 00000080:00200000:0.0:1789760792.538461:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80805888, bytes: 4096 00000080:00200000:0.0:1789760792.538472:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.538481:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.538483:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80805888, 80809984) 0 0 00000080:00200000:0.0:1789760792.538488:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.538505:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19728, 19728] 00020000:00200000:0.0:1789760792.538514:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80805888 chunk: [80805888, 80809984) 80809984, 4096 00020000:00200000:0.0:1789760792.538518:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.538539:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19728, 19728] 00000008:00010000:0.0:1789760792.538557:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.538563:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.538575:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80805888, 80809984) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.538599:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80805888=0x4d10000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.539018:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.539022:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80809984 00000020:00010000:0.0:1789760792.539032:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.539119:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.539123:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.539127:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.539129:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.539139:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.539148:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.539165:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760792.539269:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9bb80 - object 0x280000401:2 offset 80805888<>80809984 00000100:00100000:1.0:1789760792.539337:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503915392:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.552392:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987035 gen 2 00000008:00010000:1.0:1789760792.552448:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760792.552457:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.552471:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.552503:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760792.552538:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503915392:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.553159:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19728, 19728] 00000080:00200000:0.0:1789760792.553178:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.553185:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.553202:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80809984 i_blocks 157856 00000020:00200000:0.0:1789760792.553206:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.553217:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.553256:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80809984, count: 4096 00000080:00200000:0.0:1789760792.553264:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80809984, bytes: 4096 00000080:00200000:0.0:1789760792.553276:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.553285:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.553288:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80809984, 80814080) 0 0 00000080:00200000:0.0:1789760792.553292:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.553620:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19729, 19729] 00020000:00200000:0.0:1789760792.553639:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80809984 chunk: [80809984, 80814080) 80814080, 4096 00020000:00200000:0.0:1789760792.553644:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.553671:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19729, 19729] 00000008:00010000:0.0:1789760792.553690:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.553697:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.553709:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80809984, 80814080) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.553722:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80809984=0x4d11000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.553790:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.553793:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80814080 00000020:00010000:0.0:1789760792.553803:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.553806:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.553810:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.553813:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.553816:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.553825:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.553832:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.553855:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760792.553962:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98700 - object 0x280000401:2 offset 80809984<>80814080 00000100:00100000:1.0:1789760792.554355:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503915520:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.567241:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987036 gen 2 00000008:00010000:1.0:1789760792.567294:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760792.567302:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.567315:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.567336:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760792.567369:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503915520:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.567472:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19729, 19729] 00000080:00200000:0.0:1789760792.567488:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.567503:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.567518:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80814080 i_blocks 157864 00000020:00200000:0.0:1789760792.567521:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.567530:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.567564:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80814080, count: 4096 00000080:00200000:0.0:1789760792.567570:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80814080, bytes: 4096 00000080:00200000:0.0:1789760792.567580:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.567598:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.567601:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80814080, 80818176) 0 0 00000080:00200000:0.0:1789760792.567605:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.567628:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19730, 19730] 00020000:00200000:0.0:1789760792.567638:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80814080 chunk: [80814080, 80818176) 80818176, 4096 00020000:00200000:0.0:1789760792.567643:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.567774:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19730, 19730] 00000008:00010000:0.0:1789760792.567797:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.567803:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.567815:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80814080, 80818176) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.567829:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80814080=0x4d12000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.567889:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.567891:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80818176 00000020:00010000:0.0:1789760792.567900:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.567903:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.567906:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.567909:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.567912:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.567920:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.567927:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.567941:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760792.568092:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98700 - object 0x280000401:2 offset 80814080<>80818176 00000100:00100000:1.0:1789760792.568254:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760792.568273:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503915648:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.582653:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987037 gen 2 00000008:00010000:1.0:1789760792.582705:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760792.582713:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.582725:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.582738:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000080:00200000:0.0:1789760792.582767:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19730, 19730] 00000100:00100000:1.0:1789760792.582771:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503915648:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.582785:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.582791:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.582806:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80818176 i_blocks 157872 00000020:00200000:0.0:1789760792.582810:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.582820:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.582855:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80818176, count: 4096 00000080:00200000:0.0:1789760792.582862:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80818176, bytes: 4096 00000080:00200000:0.0:1789760792.582872:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.582882:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.582885:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80818176, 80822272) 0 0 00000080:00200000:0.0:1789760792.582889:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.582906:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19731, 19731] 00020000:00200000:0.0:1789760792.582915:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80818176 chunk: [80818176, 80822272) 80822272, 4096 00020000:00200000:0.0:1789760792.582921:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.582941:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19731, 19731] 00000008:00010000:0.0:1789760792.582959:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.582965:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.582976:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80818176, 80822272) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.582990:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80818176=0x4d13000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.583091:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.583094:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80822272 00000020:00010000:0.0:1789760792.583103:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.583106:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.583109:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.583113:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.583116:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.583124:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.583131:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.583148:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760792.583251:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9bb80 - object 0x280000401:2 offset 80818176<>80822272 00000100:00100000:1.0:1789760792.583280:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503915776:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.596698:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987038 gen 2 00000008:00010000:1.0:1789760792.596749:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760792.596757:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.596769:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.596794:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760792.596826:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503915776:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.597067:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19731, 19731] 00000080:00200000:2.0:1789760792.597085:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.597092:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.597108:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80822272 i_blocks 157880 00000020:00200000:2.0:1789760792.597112:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.597122:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.597159:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80822272, count: 4096 00000080:00200000:2.0:1789760792.597165:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80822272, bytes: 4096 00000080:00200000:2.0:1789760792.597177:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.597186:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.597189:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80822272, 80826368) 0 0 00000080:00200000:2.0:1789760792.597194:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.597213:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19732, 19732] 00020000:00200000:2.0:1789760792.597222:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80822272 chunk: [80822272, 80826368) 80826368, 4096 00020000:00200000:2.0:1789760792.597227:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.597249:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19732, 19732] 00000008:00010000:2.0:1789760792.597267:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.597274:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.597285:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80822272, 80826368) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.597299:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80822272=0x4d14000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.597369:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.597372:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80826368 00000020:00010000:2.0:1789760792.597380:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.597384:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.597387:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.597391:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.597394:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.597402:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.597409:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.597427:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.597738:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80822272<>80826368 00000100:00100000:3.0:1789760792.597823:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503915904:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.613905:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987039 gen 2 00000008:00010000:2.0:1789760792.613955:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.613963:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.613973:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.614004:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.614064:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503915904:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760792.614194:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19732, 19732] 00000080:00200000:1.0:1789760792.614211:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.614217:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.614234:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80826368 i_blocks 157888 00000020:00200000:1.0:1789760792.614238:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.614249:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.614287:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80826368, count: 4096 00000080:00200000:1.0:1789760792.614294:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80826368, bytes: 4096 00000080:00200000:1.0:1789760792.614957:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.614971:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760792.614973:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80826368, 80830464) 0 0 00000080:00200000:1.0:1789760792.614980:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.614999:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19733, 19733] 00020000:00200000:1.0:1789760792.615009:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80826368 chunk: [80826368, 80830464) 80830464, 4096 00020000:00200000:1.0:1789760792.615013:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.615068:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19733, 19733] 00000008:00010000:1.0:1789760792.615091:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.615097:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760792.615109:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80826368, 80830464) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.615123:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80826368=0x4d15000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.615191:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.615195:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80830464 00000020:00010000:1.0:1789760792.615203:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.615206:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.615209:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.615213:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.615215:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.615225:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.615231:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.615255:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760792.616545:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 80826368<>80830464 00000100:00100000:1.0:1789760792.616630:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760792.616811:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503916032:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.637959:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987040 gen 2 00000008:00010000:1.0:1789760792.638013:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d583c0 00000020:00200000:1.0:1789760792.638022:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.638066:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.638090:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d583c0 00000100:00100000:1.0:1789760792.638122:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503916032:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.639372:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19733, 19733] 00000080:00200000:0.0:1789760792.639393:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.639400:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.639428:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80830464 i_blocks 157896 00000020:00200000:0.0:1789760792.639434:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.639447:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.639478:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80830464, count: 4096 00000080:00200000:0.0:1789760792.639485:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80830464, bytes: 4096 00000080:00200000:0.0:1789760792.639497:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.639506:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.639508:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80830464, 80834560) 0 0 00000080:00200000:0.0:1789760792.639512:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.639528:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19734, 19734] 00020000:00200000:0.0:1789760792.639535:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80830464 chunk: [80830464, 80834560) 80834560, 4096 00020000:00200000:0.0:1789760792.639540:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.639560:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19734, 19734] 00000008:00010000:0.0:1789760792.639577:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.639594:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.639606:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80830464, 80834560) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.639631:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80830464=0x4d16000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.639695:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.639703:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80834560 00000020:00010000:0.0:1789760792.639712:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.639715:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.639719:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.639723:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.639726:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.639736:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.639742:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.639757:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760792.639871:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99f80 - object 0x280000401:2 offset 80830464<>80834560 00000100:00100000:0.0:1789760792.639917:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760792.639941:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503916160:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760792.658568:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987041 gen 2 00000008:00010000:0.0:1789760792.658632:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:0.0:1789760792.658650:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.658663:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.658689:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:0.0:1789760792.658723:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503916160:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.658854:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19734, 19734] 00000080:00200000:2.0:1789760792.658872:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.658879:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.658895:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80834560 i_blocks 157904 00000020:00200000:2.0:1789760792.658899:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.658909:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.658944:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80834560, count: 4096 00000080:00200000:2.0:1789760792.658952:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80834560, bytes: 4096 00000080:00200000:2.0:1789760792.658967:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.658976:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.658978:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80834560, 80838656) 0 0 00000080:00200000:2.0:1789760792.658982:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.658998:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19735, 19735] 00020000:00200000:2.0:1789760792.659008:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80834560 chunk: [80834560, 80838656) 80838656, 4096 00020000:00200000:2.0:1789760792.659013:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.659066:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19735, 19735] 00000008:00010000:2.0:1789760792.659088:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.659094:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.659106:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80834560, 80838656) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.659120:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80834560=0x4d17000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.659188:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.659191:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80838656 00000020:00010000:2.0:1789760792.659199:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.659203:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.659206:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.659210:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.659213:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.659220:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.659226:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.659242:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.659471:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80834560<>80838656 00000100:00100000:2.0:1789760792.659520:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760792.659539:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503916288:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.672962:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987042 gen 2 00000008:00010000:2.0:1789760792.673013:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.673022:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.673065:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.673148:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.673179:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503916288:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.673481:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19735, 19735] 00000080:00200000:0.0:1789760792.673543:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.673550:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.673566:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80838656 i_blocks 157912 00000020:00200000:0.0:1789760792.673570:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.673589:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.673687:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80838656, count: 4096 00000080:00200000:0.0:1789760792.673695:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80838656, bytes: 4096 00000080:00200000:0.0:1789760792.673707:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.673716:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.673718:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80838656, 80842752) 0 0 00000080:00200000:0.0:1789760792.673722:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.673739:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19736, 19736] 00020000:00200000:0.0:1789760792.673748:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80838656 chunk: [80838656, 80842752) 80842752, 4096 00020000:00200000:0.0:1789760792.673753:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.673772:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19736, 19736] 00000008:00010000:0.0:1789760792.673791:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.673796:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.673806:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80838656, 80842752) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.673820:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80838656=0x4d18000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.673879:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.673882:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80842752 00000020:00010000:0.0:1789760792.673890:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.673895:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.673898:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.673901:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.673904:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.673912:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.673918:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.673938:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760792.674326:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99f80 - object 0x280000401:2 offset 80838656<>80842752 00000100:00100000:0.0:1789760792.674447:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503916416:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760792.684739:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987043 gen 2 00000008:00010000:0.0:1789760792.684797:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:0.0:1789760792.684806:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.684817:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.684845:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:0.0:1789760792.684877:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503916416:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760792.685003:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19736, 19736] 00000080:00200000:1.0:1789760792.685020:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.685027:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.685117:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80842752 i_blocks 157920 00000020:00200000:1.0:1789760792.685122:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.685133:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.685170:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80842752, count: 4096 00000080:00200000:1.0:1789760792.685178:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80842752, bytes: 4096 00000080:00200000:1.0:1789760792.685190:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.685199:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760792.685202:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80842752, 80846848) 0 0 00000080:00200000:1.0:1789760792.685206:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.685224:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19737, 19737] 00020000:00200000:1.0:1789760792.685233:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80842752 chunk: [80842752, 80846848) 80846848, 4096 00020000:00200000:1.0:1789760792.685238:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.685260:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19737, 19737] 00000008:00010000:1.0:1789760792.685280:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.685287:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760792.685299:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80842752, 80846848) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.685313:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80842752=0x4d19000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.686501:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.686509:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80846848 00000020:00010000:1.0:1789760792.686520:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.686524:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.686528:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.686532:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.686535:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.686546:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.686554:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.686578:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760792.686705:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 80842752<>80846848 00000100:00100000:0.0:1789760792.686925:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503916544:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760792.705573:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987044 gen 2 00000008:00010000:0.0:1789760792.705633:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d583c0 00000020:00200000:0.0:1789760792.705640:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.706591:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.706681:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d583c0 00000100:00100000:0.0:1789760792.706714:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503916544:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760792.711707:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19737, 19737] 00000080:00200000:1.0:1789760792.711734:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.711741:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.711761:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80846848 i_blocks 157928 00000020:00200000:1.0:1789760792.711766:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.711782:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.711817:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80846848, count: 4096 00000080:00200000:1.0:1789760792.711826:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80846848, bytes: 4096 00000080:00200000:1.0:1789760792.711841:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.711851:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760792.711854:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80846848, 80850944) 0 0 00000080:00200000:1.0:1789760792.711858:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.711876:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19738, 19738] 00020000:00200000:1.0:1789760792.711885:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80846848 chunk: [80846848, 80850944) 80850944, 4096 00020000:00200000:1.0:1789760792.711890:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.711912:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19738, 19738] 00000008:00010000:1.0:1789760792.711932:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.711938:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760792.711950:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80846848, 80850944) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.711965:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80846848=0x4d1a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.712073:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.712077:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80850944 00000020:00010000:1.0:1789760792.712085:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.712089:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.712092:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.712095:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.712098:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.712109:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.712115:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.712140:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760792.712258:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 80846848<>80850944 00000100:00100000:0.0:1789760792.712777:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503916672:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760792.724904:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987045 gen 2 00000008:00010000:0.0:1789760792.724962:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:0.0:1789760792.724971:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.724982:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.725120:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:0.0:1789760792.725148:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503916672:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.725866:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19738, 19738] 00000080:00200000:2.0:1789760792.725885:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.725892:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.726020:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80850944 i_blocks 157936 00000020:00200000:2.0:1789760792.726026:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.726063:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.726098:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80850944, count: 4096 00000080:00200000:2.0:1789760792.726106:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80850944, bytes: 4096 00000080:00200000:2.0:1789760792.726119:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.726127:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.726130:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80850944, 80855040) 0 0 00000080:00200000:2.0:1789760792.726134:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.726151:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19739, 19739] 00020000:00200000:2.0:1789760792.726160:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80850944 chunk: [80850944, 80855040) 80855040, 4096 00020000:00200000:2.0:1789760792.726165:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.726185:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19739, 19739] 00000008:00010000:2.0:1789760792.726203:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.726208:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.726220:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80850944, 80855040) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.726234:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80850944=0x4d1b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.726291:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.726294:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80855040 00000020:00010000:2.0:1789760792.726365:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.726370:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.726374:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.726377:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.726380:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.726507:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.726515:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.726538:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.726722:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80850944<>80855040 00000100:00100000:3.0:1789760792.726812:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503916800:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.739554:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987046 gen 2 00000008:00010000:2.0:1789760792.739617:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.739734:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.739746:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.739770:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.739801:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503916800:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.740072:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19739, 19739] 00000080:00200000:0.0:1789760792.740089:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.740095:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.740112:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80855040 i_blocks 157944 00000020:00200000:0.0:1789760792.740117:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.740127:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.740160:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80855040, count: 4096 00000080:00200000:0.0:1789760792.740167:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80855040, bytes: 4096 00000080:00200000:0.0:1789760792.740178:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.740187:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.740190:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80855040, 80859136) 0 0 00000080:00200000:0.0:1789760792.740195:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.740213:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19740, 19740] 00020000:00200000:0.0:1789760792.740224:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80855040 chunk: [80855040, 80859136) 80859136, 4096 00020000:00200000:0.0:1789760792.740229:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.740251:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19740, 19740] 00000008:00010000:0.0:1789760792.740269:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.740275:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.740286:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80855040, 80859136) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.740300:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80855040=0x4d1c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.740369:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.740372:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80859136 00000020:00010000:0.0:1789760792.740381:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.740385:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.740388:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.740392:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.740394:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.740594:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.740606:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.740699:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760792.740815:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99f80 - object 0x280000401:2 offset 80855040<>80859136 00000100:00100000:0.0:1789760792.740866:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760792.740889:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503916928:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760792.753161:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987047 gen 2 00000008:00010000:0.0:1789760792.753212:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:0.0:1789760792.753221:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.753231:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.753319:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:0.0:1789760792.753350:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503916928:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.753421:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19740, 19740] 00000080:00200000:2.0:1789760792.753437:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.753443:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.753460:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80859136 i_blocks 157952 00000020:00200000:2.0:1789760792.753464:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.753475:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.753509:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80859136, count: 4096 00000080:00200000:2.0:1789760792.753516:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80859136, bytes: 4096 00000080:00200000:2.0:1789760792.753526:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.753534:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.753537:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80859136, 80863232) 0 0 00000080:00200000:2.0:1789760792.753541:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.753559:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19741, 19741] 00020000:00200000:2.0:1789760792.753602:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80859136 chunk: [80859136, 80863232) 80863232, 4096 00020000:00200000:2.0:1789760792.753609:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.753700:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19741, 19741] 00000008:00010000:2.0:1789760792.753720:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.753725:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.753736:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80859136, 80863232) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.753750:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80859136=0x4d1d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.753811:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.753814:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80863232 00000020:00010000:2.0:1789760792.753823:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.753828:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.753830:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.753834:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.753837:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.753845:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.753852:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.753868:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.753973:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80859136<>80863232 00000100:00100000:2.0:1789760792.754025:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760792.754069:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503917056:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.769439:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987048 gen 2 00000008:00010000:2.0:1789760792.769508:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.769516:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.769529:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.769555:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.769594:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503917056:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.769988:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19741, 19741] 00000080:00200000:0.0:1789760792.770009:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.770015:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.770063:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80863232 i_blocks 157960 00000020:00200000:0.0:1789760792.770068:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.770080:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.770116:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80863232, count: 4096 00000080:00200000:0.0:1789760792.770123:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80863232, bytes: 4096 00000080:00200000:0.0:1789760792.770135:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.770143:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.770145:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80863232, 80867328) 0 0 00000080:00200000:0.0:1789760792.770150:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.770167:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19742, 19742] 00020000:00200000:0.0:1789760792.770176:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80863232 chunk: [80863232, 80867328) 80867328, 4096 00020000:00200000:0.0:1789760792.770181:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.770213:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19742, 19742] 00000008:00010000:0.0:1789760792.770231:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.770237:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.770249:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80863232, 80867328) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.770264:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80863232=0x4d1e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.770337:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.770341:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80867328 00000020:00010000:0.0:1789760792.770350:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.770353:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.770356:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.770359:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.770363:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.770371:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.770378:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.770397:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760792.770516:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99f80 - object 0x280000401:2 offset 80863232<>80867328 00000100:00100000:1.0:1789760792.771195:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503917184:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.782731:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987049 gen 2 00000008:00010000:1.0:1789760792.782788:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760792.782797:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.782809:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.782833:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760792.782867:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503917184:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760792.783069:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19742, 19742] 00000080:00200000:3.0:1789760792.783086:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.783092:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760792.783109:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80867328 i_blocks 157968 00000020:00200000:3.0:1789760792.783113:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760792.783123:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760792.783160:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80867328, count: 4096 00000080:00200000:3.0:1789760792.783167:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80867328, bytes: 4096 00000080:00200000:3.0:1789760792.783179:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760792.783188:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760792.783200:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80867328, 80871424) 0 0 00000080:00200000:3.0:1789760792.783205:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760792.783223:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19743, 19743] 00020000:00200000:3.0:1789760792.783232:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80867328 chunk: [80867328, 80871424) 80871424, 4096 00020000:00200000:3.0:1789760792.783236:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760792.783256:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19743, 19743] 00000008:00010000:3.0:1789760792.783274:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760792.783281:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760792.783328:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80867328, 80871424) append 0 append lockpos 0 00000080:00200000:3.0:1789760792.783342:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80867328=0x4d1f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760792.783407:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760792.783409:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80871424 00000020:00010000:3.0:1789760792.783418:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760792.783423:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.783426:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.783430:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760792.783434:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760792.783442:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760792.783449:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760792.783467:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1540 00000008:00100000:3.0:1789760792.783916:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070df80 - object 0x280000401:2 offset 80867328<>80871424 00000100:00100000:3.0:1789760792.783968:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760792.783987:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503917312:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.800709:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987050 gen 2 00000008:00010000:2.0:1789760792.800761:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1540 00000020:00200000:2.0:1789760792.800770:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.800781:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.800847:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1540 00000100:00100000:2.0:1789760792.800880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503917312:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760792.800977:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19743, 19743] 00000080:00200000:3.0:1789760792.800992:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.800998:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.801117:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80871424 i_blocks 157976 00000020:00200000:2.0:1789760792.801122:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.801133:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.801169:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80871424, count: 4096 00000080:00200000:2.0:1789760792.801177:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80871424, bytes: 4096 00000080:00200000:2.0:1789760792.801187:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.801197:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.801200:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80871424, 80875520) 0 0 00000080:00200000:2.0:1789760792.801204:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.801222:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19744, 19744] 00020000:00200000:2.0:1789760792.801230:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80871424 chunk: [80871424, 80875520) 80875520, 4096 00020000:00200000:2.0:1789760792.801235:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.801254:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19744, 19744] 00000008:00010000:2.0:1789760792.801272:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.801277:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.801287:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80871424, 80875520) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.801300:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80871424=0x4d20000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.801433:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.801436:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80875520 00000020:00010000:2.0:1789760792.801445:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.801448:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.801451:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.801454:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.801457:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.801465:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.801471:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.801486:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.801600:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80871424<>80875520 00000100:00100000:3.0:1789760792.801727:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503917440:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.814477:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987051 gen 2 00000008:00010000:2.0:1789760792.814527:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.814536:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.814546:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.814619:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.814653:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503917440:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.814738:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19744, 19744] 00000080:00200000:0.0:1789760792.814754:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.814760:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.814775:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80875520 i_blocks 157984 00000020:00200000:0.0:1789760792.814780:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.814789:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.814910:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80875520, count: 4096 00000080:00200000:0.0:1789760792.814919:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80875520, bytes: 4096 00000080:00200000:0.0:1789760792.814930:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.814938:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.814940:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80875520, 80879616) 0 0 00000080:00200000:0.0:1789760792.814945:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.814960:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19745, 19745] 00020000:00200000:0.0:1789760792.814969:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80875520 chunk: [80875520, 80879616) 80879616, 4096 00020000:00200000:0.0:1789760792.814973:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.814994:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19745, 19745] 00000008:00010000:0.0:1789760792.815012:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.815018:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.815028:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80875520, 80879616) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.815064:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80875520=0x4d21000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.815121:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.815124:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80879616 00000020:00010000:0.0:1789760792.815131:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.815134:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.815137:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.815141:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.815145:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.815152:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.815160:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.815176:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760792.815277:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99880 - object 0x280000401:2 offset 80875520<>80879616 00000100:00100000:1.0:1789760792.815352:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503917568:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.827889:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987052 gen 2 00000008:00010000:1.0:1789760792.827943:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760792.827952:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.827965:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.828079:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760792.828114:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503917568:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.834436:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19745, 19745] 00000080:00200000:0.0:1789760792.834465:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.834472:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.834492:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80879616 i_blocks 157992 00000020:00200000:0.0:1789760792.834497:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.834514:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.834549:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80879616, count: 4096 00000080:00200000:0.0:1789760792.834559:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80879616, bytes: 4096 00000080:00200000:0.0:1789760792.834574:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.834597:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.834600:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80879616, 80883712) 0 0 00000080:00200000:0.0:1789760792.834605:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.834623:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19746, 19746] 00020000:00200000:0.0:1789760792.834632:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80879616 chunk: [80879616, 80883712) 80883712, 4096 00020000:00200000:0.0:1789760792.834636:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.834660:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19746, 19746] 00000008:00010000:0.0:1789760792.834680:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.834686:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.834697:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80879616, 80883712) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.834712:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80879616=0x4d22000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.834785:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.834788:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80883712 00000020:00010000:0.0:1789760792.834797:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.834800:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.834804:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.834807:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.834810:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.834818:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.834826:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.834854:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760792.834978:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 80879616<>80883712 00000100:00100000:1.0:1789760792.835309:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760792.835338:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503917696:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.848001:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987053 gen 2 00000008:00010000:1.0:1789760792.848098:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760792.848241:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.848255:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.848268:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760792.848301:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503917696:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.849007:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19746, 19746] 00000080:00200000:0.0:1789760792.849024:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.849031:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.849084:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80883712 i_blocks 158000 00000020:00200000:0.0:1789760792.849088:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.849098:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.849133:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80883712, count: 4096 00000080:00200000:0.0:1789760792.849140:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80883712, bytes: 4096 00000080:00200000:0.0:1789760792.849152:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.849162:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.849164:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80883712, 80887808) 0 0 00000080:00200000:0.0:1789760792.849169:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.849185:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19747, 19747] 00020000:00200000:0.0:1789760792.849194:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80883712 chunk: [80883712, 80887808) 80887808, 4096 00020000:00200000:0.0:1789760792.849199:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.849219:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19747, 19747] 00000008:00010000:0.0:1789760792.849238:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.849244:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.849255:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80883712, 80887808) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.849271:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80883712=0x4d23000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.849391:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.849394:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80887808 00000020:00010000:0.0:1789760792.849403:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.849407:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.849410:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.849413:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.849416:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.849424:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.849430:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.849451:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760792.849696:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98a80 - object 0x280000401:2 offset 80883712<>80887808 00000100:00100000:1.0:1789760792.849772:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503917824:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.862337:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987054 gen 2 00000008:00010000:1.0:1789760792.862391:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760792.862399:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.862412:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.862438:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760792.862472:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503917824:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.862759:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19747, 19747] 00000080:00200000:2.0:1789760792.862776:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.862783:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.862801:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80887808 i_blocks 158008 00000020:00200000:2.0:1789760792.862805:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.862817:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.862852:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80887808, count: 4096 00000080:00200000:2.0:1789760792.862859:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80887808, bytes: 4096 00000080:00200000:2.0:1789760792.862871:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.862879:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.862881:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80887808, 80891904) 0 0 00000080:00200000:2.0:1789760792.862886:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.862904:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19748, 19748] 00020000:00200000:2.0:1789760792.862913:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80887808 chunk: [80887808, 80891904) 80891904, 4096 00020000:00200000:2.0:1789760792.862917:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.862936:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19748, 19748] 00000008:00010000:2.0:1789760792.862954:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.862961:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.862971:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80887808, 80891904) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.862985:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80887808=0x4d24000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.863067:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.863070:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80891904 00000020:00010000:2.0:1789760792.863078:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.863081:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.863085:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.863088:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.863091:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.863099:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.863106:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.863123:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.863234:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80887808<>80891904 00000100:00100000:3.0:1789760792.863311:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503917952:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.875104:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987055 gen 2 00000008:00010000:2.0:1789760792.875157:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.875165:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.875176:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.875263:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.875294:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503917952:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.875439:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19748, 19748] 00000080:00200000:0.0:1789760792.875455:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.875460:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.875477:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80891904 i_blocks 158016 00000020:00200000:0.0:1789760792.875480:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.875498:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.875534:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80891904, count: 4096 00000080:00200000:0.0:1789760792.875542:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80891904, bytes: 4096 00000080:00200000:0.0:1789760792.875553:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.875561:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.875564:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80891904, 80896000) 0 0 00000080:00200000:0.0:1789760792.875568:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.875595:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19749, 19749] 00020000:00200000:0.0:1789760792.875605:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80891904 chunk: [80891904, 80896000) 80896000, 4096 00020000:00200000:0.0:1789760792.875610:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.875748:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19749, 19749] 00000008:00010000:0.0:1789760792.875769:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.875774:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.875786:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80891904, 80896000) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.875802:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80891904=0x4d25000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.875864:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.875867:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80896000 00000020:00010000:0.0:1789760792.875877:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.875881:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.875884:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.875888:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.875891:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.875900:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.876030:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.876150:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760792.876258:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98000 - object 0x280000401:2 offset 80891904<>80896000 00000100:00100000:1.0:1789760792.876391:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503918080:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.885468:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987056 gen 2 00000008:00010000:1.0:1789760792.885519:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760792.885528:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.885541:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.885612:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760792.885646:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503918080:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.885790:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19749, 19749] 00000080:00200000:0.0:1789760792.885807:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.885814:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.885829:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80896000 i_blocks 158024 00000020:00200000:0.0:1789760792.885833:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.885843:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.885876:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80896000, count: 4096 00000080:00200000:0.0:1789760792.885883:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80896000, bytes: 4096 00000080:00200000:0.0:1789760792.885894:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.885903:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.885905:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80896000, 80900096) 0 0 00000080:00200000:0.0:1789760792.885910:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.885926:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19750, 19750] 00020000:00200000:0.0:1789760792.885935:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80896000 chunk: [80896000, 80900096) 80900096, 4096 00020000:00200000:0.0:1789760792.885939:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.885959:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19750, 19750] 00000008:00010000:0.0:1789760792.885977:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.885983:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.885993:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80896000, 80900096) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.886008:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80896000=0x4d26000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.886331:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.886335:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80900096 00000020:00010000:0.0:1789760792.886344:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.886347:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.886351:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.886354:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.886357:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.886365:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.886373:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.886391:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760792.886484:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a680 - object 0x280000401:2 offset 80896000<>80900096 00000100:00100000:1.0:1789760792.886554:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503918208:192.168.203.158@tcp:4:dd.0 00010000:00010000:0.0:1789760792.896945:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760792.903872:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987057 gen 2 00000008:00010000:1.0:1789760792.903925:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760792.903933:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.903945:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.903970:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760792.904003:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503918208:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.904144:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19750, 19750] 00000080:00200000:2.0:1789760792.904159:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.904166:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.904183:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80900096 i_blocks 158032 00000020:00200000:2.0:1789760792.904188:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.904198:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.904231:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80900096, count: 4096 00000080:00200000:2.0:1789760792.904238:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80900096, bytes: 4096 00000080:00200000:2.0:1789760792.904250:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.904258:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.904260:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80900096, 80904192) 0 0 00000080:00200000:2.0:1789760792.904265:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.904280:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19751, 19751] 00020000:00200000:2.0:1789760792.904288:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80900096 chunk: [80900096, 80904192) 80904192, 4096 00020000:00200000:2.0:1789760792.904293:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.904384:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19751, 19751] 00000008:00010000:2.0:1789760792.904405:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.904411:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.904424:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80900096, 80904192) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.904438:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80900096=0x4d27000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.904495:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.904498:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80904192 00000020:00010000:2.0:1789760792.904507:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.904510:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.904514:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.904519:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.904522:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.904530:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.904538:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.904557:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.904686:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80900096<>80904192 00000100:00100000:3.0:1789760792.904768:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503918336:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.913311:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987058 gen 2 00000008:00010000:2.0:1789760792.913365:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.913374:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.913385:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.913458:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.913491:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503918336:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.913819:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19751, 19751] 00000080:00200000:0.0:1789760792.913863:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.913869:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.913885:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80904192 i_blocks 158040 00000020:00200000:0.0:1789760792.913888:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.913899:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.913935:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80904192, count: 4096 00000080:00200000:0.0:1789760792.913943:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80904192, bytes: 4096 00000080:00200000:0.0:1789760792.913954:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.913964:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.913966:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80904192, 80908288) 0 0 00000080:00200000:0.0:1789760792.913971:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.913986:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19752, 19752] 00020000:00200000:0.0:1789760792.913995:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80904192 chunk: [80904192, 80908288) 80908288, 4096 00020000:00200000:0.0:1789760792.913999:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.914019:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19752, 19752] 00000008:00010000:0.0:1789760792.914237:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.914244:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.914259:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80904192, 80908288) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.914273:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80904192=0x4d28000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.914412:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.914416:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80908288 00000020:00010000:0.0:1789760792.914425:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.914429:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.914432:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.914436:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.914439:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.914448:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.914454:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.914474:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760792.914953:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b480 - object 0x280000401:2 offset 80904192<>80908288 00000100:00100000:0.0:1789760792.915015:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503918464:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.926020:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987059 gen 2 00000008:00010000:1.0:1789760792.926107:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760792.926115:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.926127:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.926139:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760792.926168:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503918464:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.926350:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19752, 19752] 00000080:00200000:0.0:1789760792.926366:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.926372:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.926388:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80908288 i_blocks 158048 00000020:00200000:0.0:1789760792.926392:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.926401:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.926435:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80908288, count: 4096 00000080:00200000:0.0:1789760792.926442:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80908288, bytes: 4096 00000080:00200000:0.0:1789760792.926451:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.926461:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.926463:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80908288, 80912384) 0 0 00000080:00200000:0.0:1789760792.926468:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.926483:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19753, 19753] 00020000:00200000:0.0:1789760792.926505:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80908288 chunk: [80908288, 80912384) 80912384, 4096 00020000:00200000:0.0:1789760792.926509:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.926531:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19753, 19753] 00000008:00010000:0.0:1789760792.926548:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.926555:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.926566:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80908288, 80912384) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.926590:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80908288=0x4d29000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.926656:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.926669:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80912384 00000020:00010000:0.0:1789760792.926678:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.926841:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.926845:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.926848:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.926851:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.926861:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.926868:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.926886:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760792.926987:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b800 - object 0x280000401:2 offset 80908288<>80912384 00000100:00100000:1.0:1789760792.927212:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760792.927232:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503918592:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.940150:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987060 gen 2 00000008:00010000:1.0:1789760792.940218:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760792.940228:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.940242:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.940256:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760792.940289:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503918592:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760792.941007:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19753, 19753] 00000080:00200000:0.0:1789760792.941026:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.941032:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.941081:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80912384 i_blocks 158056 00000020:00200000:0.0:1789760792.941086:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.941096:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.941128:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80912384, count: 4096 00000080:00200000:0.0:1789760792.941134:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80912384, bytes: 4096 00000080:00200000:0.0:1789760792.941147:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.941156:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760792.941158:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80912384, 80916480) 0 0 00000080:00200000:0.0:1789760792.941171:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.941186:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19754, 19754] 00020000:00200000:0.0:1789760792.941195:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80912384 chunk: [80912384, 80916480) 80916480, 4096 00020000:00200000:0.0:1789760792.941199:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.941218:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19754, 19754] 00000008:00010000:0.0:1789760792.941234:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.941239:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760792.941250:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80912384, 80916480) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.941263:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80912384=0x4d2a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.941326:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.941328:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80916480 00000020:00010000:0.0:1789760792.941336:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.941339:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.941342:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.941345:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.941348:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.941356:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.941361:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.941376:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760792.941471:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9aa00 - object 0x280000401:2 offset 80912384<>80916480 00000100:00100000:1.0:1789760792.941598:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503918720:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.952102:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987061 gen 2 00000008:00010000:1.0:1789760792.952153:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760792.952162:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.952175:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.952200:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760792.952235:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503918720:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.952353:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19754, 19754] 00000080:00200000:2.0:1789760792.952370:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.952376:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.952394:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80916480 i_blocks 158064 00000020:00200000:2.0:1789760792.952398:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.952408:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.952444:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80916480, count: 4096 00000080:00200000:2.0:1789760792.952451:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80916480, bytes: 4096 00000080:00200000:2.0:1789760792.952463:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.952471:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.952473:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80916480, 80920576) 0 0 00000080:00200000:2.0:1789760792.952478:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.952496:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19755, 19755] 00020000:00200000:2.0:1789760792.952504:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80916480 chunk: [80916480, 80920576) 80920576, 4096 00020000:00200000:2.0:1789760792.952508:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.952527:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19755, 19755] 00000008:00010000:2.0:1789760792.952546:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.952552:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.952565:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80916480, 80920576) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.952591:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80916480=0x4d2b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.952653:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.952656:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80920576 00000020:00010000:2.0:1789760792.952665:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.952668:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.952672:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.952676:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.952693:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.952702:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.952709:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.952727:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.952833:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80916480<>80920576 00000100:00100000:2.0:1789760792.952883:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760792.952901:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503918848:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.964378:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987062 gen 2 00000008:00010000:2.0:1789760792.964439:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.964447:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.964459:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.964497:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.964530:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503918848:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760792.965074:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19755, 19755] 00000080:00200000:1.0:1789760792.965093:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.965100:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.965130:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80920576 i_blocks 158072 00000020:00200000:1.0:1789760792.965134:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.965145:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.965181:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80920576, count: 4096 00000080:00200000:1.0:1789760792.965189:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80920576, bytes: 4096 00000080:00200000:1.0:1789760792.965202:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.965212:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760792.965215:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80920576, 80924672) 0 0 00000080:00200000:1.0:1789760792.965220:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.965238:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19756, 19756] 00020000:00200000:1.0:1789760792.965247:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80920576 chunk: [80920576, 80924672) 80924672, 4096 00020000:00200000:1.0:1789760792.965252:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.965274:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19756, 19756] 00000008:00010000:1.0:1789760792.965291:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.965297:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760792.966514:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80920576, 80924672) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.966538:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80920576=0x4d2c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.966647:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.966651:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80924672 00000020:00010000:1.0:1789760792.966659:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.966662:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.966665:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.966669:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.966671:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.966968:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.966981:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.967011:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760792.967171:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 80920576<>80924672 00000100:00100000:1.0:1789760792.967214:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760792.967239:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503918976:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760792.978110:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987063 gen 2 00000008:00010000:1.0:1789760792.978162:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:1.0:1789760792.978171:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.978182:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.978207:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:1.0:1789760792.978239:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503918976:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760792.979057:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19756, 19756] 00000080:00200000:2.0:1789760792.979078:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.979084:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.979101:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80924672 i_blocks 158080 00000020:00200000:2.0:1789760792.979105:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.979117:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.979152:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80924672, count: 4096 00000080:00200000:2.0:1789760792.979159:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80924672, bytes: 4096 00000080:00200000:2.0:1789760792.979173:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.979182:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760792.979185:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80924672, 80928768) 0 0 00000080:00200000:2.0:1789760792.979189:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.979208:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19757, 19757] 00020000:00200000:2.0:1789760792.979217:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80924672 chunk: [80924672, 80928768) 80928768, 4096 00020000:00200000:2.0:1789760792.979223:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.979244:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19757, 19757] 00000008:00010000:2.0:1789760792.979264:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.979271:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760792.979283:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80924672, 80928768) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.979298:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80924672=0x4d2d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.979364:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.979367:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80928768 00000020:00010000:2.0:1789760792.979375:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.979378:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.979382:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.979386:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c3062b78 (ffff9018d1a6ebd8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.979388:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018d1a6ebd8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.979398:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.979404:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.979422:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760792.979548:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80924672<>80928768 00000100:00100000:3.0:1789760792.979828:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760792.979850:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503919104:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760792.989849:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987064 gen 2 00000008:00010000:2.0:1789760792.989899:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760792.989907:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.989919:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.989946:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760792.989975:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503919104:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760792.990102:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19757, 19757] 00000080:00200000:3.0:1789760792.990118:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.990124:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760792.990140:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80928768 i_blocks 158088 00000020:00200000:3.0:1789760792.990145:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760792.990155:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760792.990218:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80928768, count: 4096 00000080:00200000:3.0:1789760792.990226:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80928768, bytes: 4096 00000080:00200000:3.0:1789760792.990238:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760792.990246:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760792.990248:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80928768, 80932864) 0 0 00000080:00200000:3.0:1789760792.990253:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760792.990271:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19758, 19758] 00020000:00200000:3.0:1789760792.990280:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80928768 chunk: [80928768, 80932864) 80932864, 4096 00020000:00200000:3.0:1789760792.990285:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760792.990380:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19758, 19758] 00000008:00010000:3.0:1789760792.990400:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760792.990406:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760792.990418:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80928768, 80932864) append 0 append lockpos 0 00000080:00200000:3.0:1789760792.990432:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80928768=0x4d2e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760792.990500:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760792.990504:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80932864 00000020:00010000:3.0:1789760792.990512:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760792.990516:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.990518:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.990522:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760792.990525:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760792.990532:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760792.990539:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760792.990556:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1f00 00000008:00100000:3.0:1789760792.990969:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ca80 - object 0x280000401:2 offset 80928768<>80932864 00000100:00100000:3.0:1789760792.991021:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760792.991067:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503919232:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760793.003272:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987065 gen 2 00000008:00010000:2.0:1789760793.003393:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1f00 00000020:00200000:2.0:1789760793.003402:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.003413:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.003502:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1f00 00000100:00100000:2.0:1789760793.003536:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503919232:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760793.003751:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19758, 19758] 00000080:00200000:3.0:1789760793.003768:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.003774:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.003791:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80932864 i_blocks 158096 00000020:00200000:3.0:1789760793.003795:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.003805:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.003842:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80932864, count: 4096 00000080:00200000:3.0:1789760793.003849:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80932864, bytes: 4096 00000080:00200000:3.0:1789760793.003863:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.003873:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760793.003875:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80932864, 80936960) 0 0 00000080:00200000:3.0:1789760793.003880:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.003896:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19759, 19759] 00020000:00200000:3.0:1789760793.003905:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80932864 chunk: [80932864, 80936960) 80936960, 4096 00020000:00200000:3.0:1789760793.003910:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.003929:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19759, 19759] 00000008:00010000:3.0:1789760793.003948:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.003954:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760793.003964:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80932864, 80936960) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.003981:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80932864=0x4d2f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.004067:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.004070:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80936960 00000020:00010000:3.0:1789760793.004078:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.004082:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.004085:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.004089:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.004091:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.004100:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.004108:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.004127:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f19c0 00000008:00100000:3.0:1789760793.004390:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070d180 - object 0x280000401:2 offset 80932864<>80936960 00000100:00100000:2.0:1789760793.004463:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503919360:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760793.013131:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987066 gen 2 00000008:00010000:2.0:1789760793.013193:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f19c0 00000020:00200000:2.0:1789760793.013202:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.013213:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.013234:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f19c0 00000100:00100000:2.0:1789760793.013265:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503919360:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760793.013962:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19759, 19759] 00000080:00200000:3.0:1789760793.013981:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.013987:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.014004:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80936960 i_blocks 158104 00000020:00200000:3.0:1789760793.014008:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.014019:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.014083:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80936960, count: 4096 00000080:00200000:3.0:1789760793.014091:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80936960, bytes: 4096 00000080:00200000:3.0:1789760793.014103:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.014113:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760793.014116:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80936960, 80941056) 0 0 00000080:00200000:3.0:1789760793.014120:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.014149:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19760, 19760] 00020000:00200000:3.0:1789760793.014159:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80936960 chunk: [80936960, 80941056) 80941056, 4096 00020000:00200000:3.0:1789760793.014163:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.014183:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19760, 19760] 00000008:00010000:3.0:1789760793.014202:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.014208:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760793.014220:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80936960, 80941056) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.014235:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80936960=0x4d30000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.014781:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.014785:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80941056 00000020:00010000:3.0:1789760793.014795:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.014798:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.014802:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.014805:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.014808:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.014816:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.014824:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.014847:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1e40 00000008:00100000:3.0:1789760793.014962:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070f100 - object 0x280000401:2 offset 80936960<>80941056 00000100:00100000:2.0:1789760793.015257:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503919488:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760793.025861:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987067 gen 2 00000008:00010000:2.0:1789760793.025928:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1e40 00000020:00200000:2.0:1789760793.025937:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.025948:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.025970:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1e40 00000100:00100000:2.0:1789760793.026000:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503919488:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760793.026510:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19760, 19760] 00000080:00200000:3.0:1789760793.026526:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.026532:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.026547:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80941056 i_blocks 158112 00000020:00200000:3.0:1789760793.026551:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.026561:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.026602:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80941056, count: 4096 00000080:00200000:3.0:1789760793.026610:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80941056, bytes: 4096 00000080:00200000:3.0:1789760793.026630:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.026641:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760793.026643:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80941056, 80945152) 0 0 00000080:00200000:3.0:1789760793.026648:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.026665:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19761, 19761] 00020000:00200000:3.0:1789760793.026673:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80941056 chunk: [80941056, 80945152) 80945152, 4096 00020000:00200000:3.0:1789760793.026678:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.026700:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19761, 19761] 00000008:00010000:3.0:1789760793.026718:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.026732:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760793.026748:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80941056, 80945152) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.026765:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80941056=0x4d31000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.026842:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.026846:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80945152 00000020:00010000:3.0:1789760793.026854:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.026857:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.026860:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.026863:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.026866:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.026873:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.026879:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.026906:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1a80 00000008:00100000:3.0:1789760793.027013:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070e300 - object 0x280000401:2 offset 80941056<>80945152 00000100:00100000:2.0:1789760793.028742:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760793.028780:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503919616:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760793.039395:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987068 gen 2 00000008:00010000:2.0:1789760793.039446:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1a80 00000020:00200000:2.0:1789760793.039455:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.039466:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.039501:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1a80 00000100:00100000:2.0:1789760793.039535:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503919616:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760793.040782:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19761, 19761] 00000080:00200000:3.0:1789760793.040803:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.040810:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.040828:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80945152 i_blocks 158120 00000020:00200000:3.0:1789760793.040833:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.040846:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.040881:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80945152, count: 4096 00000080:00200000:3.0:1789760793.040888:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80945152, bytes: 4096 00000080:00200000:3.0:1789760793.040901:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.040912:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760793.040914:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80945152, 80949248) 0 0 00000080:00200000:3.0:1789760793.040920:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.040936:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19762, 19762] 00020000:00200000:3.0:1789760793.040944:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80945152 chunk: [80945152, 80949248) 80949248, 4096 00020000:00200000:3.0:1789760793.040949:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.040971:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19762, 19762] 00000008:00010000:3.0:1789760793.040990:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.040997:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760793.041009:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80945152, 80949248) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.041024:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80945152=0x4d32000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.041115:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.041119:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80949248 00000020:00010000:3.0:1789760793.041128:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.041132:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.041135:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.041216:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.041219:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.041228:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.041235:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.041253:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f13c0 00000008:00100000:3.0:1789760793.041719:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070e680 - object 0x280000401:2 offset 80945152<>80949248 00000100:00100000:2.0:1789760793.041911:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503919744:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760793.051893:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987069 gen 2 00000008:00010000:2.0:1789760793.051947:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f13c0 00000020:00200000:2.0:1789760793.051955:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.051968:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.051992:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f13c0 00000100:00100000:2.0:1789760793.052025:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503919744:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760793.052336:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19762, 19762] 00000080:00200000:3.0:1789760793.052352:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.052359:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.052374:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80949248 i_blocks 158128 00000020:00200000:3.0:1789760793.052379:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.052388:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.052423:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80949248, count: 4096 00000080:00200000:3.0:1789760793.052430:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80949248, bytes: 4096 00000080:00200000:3.0:1789760793.052440:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.052450:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760793.052452:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80949248, 80953344) 0 0 00000080:00200000:3.0:1789760793.052458:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.052473:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19763, 19763] 00020000:00200000:3.0:1789760793.052484:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80949248 chunk: [80949248, 80953344) 80953344, 4096 00020000:00200000:3.0:1789760793.052499:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.052521:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19763, 19763] 00000008:00010000:3.0:1789760793.052705:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.052712:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760793.052726:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80949248, 80953344) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.052741:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80949248=0x4d33000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.052799:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.052803:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80953344 00000020:00010000:3.0:1789760793.052811:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.052815:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.052818:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.052821:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.052824:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.052832:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.052840:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.052860:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1d80 00000008:00100000:3.0:1789760793.052965:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070c380 - object 0x280000401:2 offset 80949248<>80953344 00000100:00100000:3.0:1789760793.053608:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760793.054109:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503919872:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760793.067517:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987070 gen 2 00000008:00010000:2.0:1789760793.067570:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1d80 00000020:00200000:2.0:1789760793.067587:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.067600:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.067630:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1d80 00000100:00100000:2.0:1789760793.067662:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503919872:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760793.067857:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19763, 19763] 00000080:00200000:0.0:1789760793.067875:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.067881:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.067896:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80953344 i_blocks 158136 00000020:00200000:0.0:1789760793.067900:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.067909:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.067944:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80953344, count: 4096 00000080:00200000:0.0:1789760793.067951:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80953344, bytes: 4096 00000080:00200000:0.0:1789760793.067961:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.067971:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760793.067973:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80953344, 80957440) 0 0 00000080:00200000:0.0:1789760793.067977:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.067994:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19764, 19764] 00020000:00200000:0.0:1789760793.068002:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80953344 chunk: [80953344, 80957440) 80957440, 4096 00020000:00200000:0.0:1789760793.068008:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.068026:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19764, 19764] 00000008:00010000:0.0:1789760793.068065:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.068072:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760793.068083:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80953344, 80957440) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.068096:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80953344=0x4d34000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.068162:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.068165:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80957440 00000020:00010000:0.0:1789760793.068173:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.068176:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.068179:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.068183:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.068185:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.068193:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.068200:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.068215:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760793.068327:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b100 - object 0x280000401:2 offset 80953344<>80957440 00000100:00100000:1.0:1789760793.068601:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760793.068631:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503920000:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760793.085313:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987071 gen 2 00000008:00010000:1.0:1789760793.085363:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760793.085372:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.085383:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.085408:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760793.085440:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503920000:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760793.085554:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19764, 19764] 00000080:00200000:0.0:1789760793.085570:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.085575:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.085598:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80957440 i_blocks 158144 00000020:00200000:0.0:1789760793.085602:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.085612:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.085644:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80957440, count: 4096 00000080:00200000:0.0:1789760793.085652:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80957440, bytes: 4096 00000080:00200000:0.0:1789760793.085661:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.085670:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760793.085673:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80957440, 80961536) 0 0 00000080:00200000:0.0:1789760793.085676:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.085693:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19765, 19765] 00020000:00200000:0.0:1789760793.085701:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80957440 chunk: [80957440, 80961536) 80961536, 4096 00020000:00200000:0.0:1789760793.085705:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.085725:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19765, 19765] 00000008:00010000:0.0:1789760793.085742:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.085749:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760793.085760:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80957440, 80961536) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.085774:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80957440=0x4d35000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.085831:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.085834:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80961536 00000020:00010000:0.0:1789760793.085842:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.085846:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.085848:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.085851:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.085854:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.085862:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.085870:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.085886:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760793.086006:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9ad80 - object 0x280000401:2 offset 80957440<>80961536 00000100:00100000:1.0:1789760793.086520:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503920128:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760793.098825:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987072 gen 2 00000008:00010000:1.0:1789760793.098892:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760793.098900:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.098915:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.098949:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760793.098983:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503920128:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760793.101182:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19765, 19765] 00000080:00200000:0.0:1789760793.101208:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.101215:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.101234:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80961536 i_blocks 158152 00000020:00200000:0.0:1789760793.101240:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.101256:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.101297:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80961536, count: 4096 00000080:00200000:0.0:1789760793.101305:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80961536, bytes: 4096 00000080:00200000:0.0:1789760793.101320:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.101329:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760793.101332:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80961536, 80965632) 0 0 00000080:00200000:0.0:1789760793.101336:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.101353:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19766, 19766] 00020000:00200000:0.0:1789760793.101362:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80961536 chunk: [80961536, 80965632) 80965632, 4096 00020000:00200000:0.0:1789760793.101367:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.101389:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19766, 19766] 00000008:00010000:0.0:1789760793.101407:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.101413:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760793.101425:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80961536, 80965632) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.101439:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80961536=0x4d36000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.101512:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.101515:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80965632 00000020:00010000:0.0:1789760793.101529:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.101533:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.101536:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.101539:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.101542:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.101551:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.101558:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.101589:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760793.101702:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98380 - object 0x280000401:2 offset 80961536<>80965632 00000100:00100000:1.0:1789760793.101850:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503920256:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760793.112109:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987073 gen 2 00000008:00010000:1.0:1789760793.112160:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760793.112168:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.112179:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.112202:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760793.112237:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503920256:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760793.113189:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19766, 19766] 00000080:00200000:0.0:1789760793.113210:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.113217:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.113234:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80965632 i_blocks 158160 00000020:00200000:0.0:1789760793.113238:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.113250:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.113285:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80965632, count: 4096 00000080:00200000:0.0:1789760793.113292:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80965632, bytes: 4096 00000080:00200000:0.0:1789760793.113303:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.113312:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760793.113315:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80965632, 80969728) 0 0 00000080:00200000:0.0:1789760793.113319:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.113336:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19767, 19767] 00020000:00200000:0.0:1789760793.113345:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80965632 chunk: [80965632, 80969728) 80969728, 4096 00020000:00200000:0.0:1789760793.113349:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.113371:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19767, 19767] 00000008:00010000:0.0:1789760793.113388:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.113395:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760793.113408:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80965632, 80969728) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.113421:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80965632=0x4d37000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.113482:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.113485:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80969728 00000020:00010000:0.0:1789760793.113558:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.113563:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.113567:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.113571:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.113574:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.113591:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.113598:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.113618:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760793.113725:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99c00 - object 0x280000401:2 offset 80965632<>80969728 00000100:00100000:1.0:1789760793.113808:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503920384:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760793.125651:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987074 gen 2 00000008:00010000:1.0:1789760793.125697:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760793.125705:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.125716:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.125738:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760793.125768:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503920384:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760793.125961:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19767, 19767] 00000080:00200000:0.0:1789760793.125976:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.125983:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.125998:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80969728 i_blocks 158168 00000020:00200000:0.0:1789760793.126001:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.126011:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.126082:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80969728, count: 4096 00000080:00200000:0.0:1789760793.126090:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80969728, bytes: 4096 00000080:00200000:0.0:1789760793.126100:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.126108:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760793.126110:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80969728, 80973824) 0 0 00000080:00200000:0.0:1789760793.126115:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.126130:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19768, 19768] 00020000:00200000:0.0:1789760793.126138:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80969728 chunk: [80969728, 80973824) 80973824, 4096 00020000:00200000:0.0:1789760793.126142:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.126160:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19768, 19768] 00000008:00010000:0.0:1789760793.126176:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.126182:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760793.126192:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80969728, 80973824) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.126205:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80969728=0x4d38000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.126259:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.126262:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80973824 00000020:00010000:0.0:1789760793.126269:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.126272:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.126275:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.126279:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.126281:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.126289:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.126294:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.126308:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760793.126403:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a300 - object 0x280000401:2 offset 80969728<>80973824 00000100:00100000:1.0:1789760793.126496:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503920512:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760793.147971:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987075 gen 2 00000008:00010000:1.0:1789760793.148027:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760793.148081:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.148095:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.148108:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760793.148211:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503920512:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760793.148611:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19768, 19768] 00000080:00200000:0.0:1789760793.148630:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.148636:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.148653:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80973824 i_blocks 158176 00000020:00200000:0.0:1789760793.148657:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.148677:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.148715:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80973824, count: 4096 00000080:00200000:0.0:1789760793.148723:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80973824, bytes: 4096 00000080:00200000:0.0:1789760793.148734:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.148743:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760793.148746:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80973824, 80977920) 0 0 00000080:00200000:0.0:1789760793.148750:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.148770:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19769, 19769] 00020000:00200000:0.0:1789760793.148779:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80973824 chunk: [80973824, 80977920) 80977920, 4096 00020000:00200000:0.0:1789760793.148784:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.148803:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19769, 19769] 00000008:00010000:0.0:1789760793.148820:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.148825:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760793.148837:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80973824, 80977920) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.148852:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80973824=0x4d39000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.148913:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.148916:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80977920 00000020:00010000:0.0:1789760793.148925:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.148928:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.148931:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.148936:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.148939:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.148947:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.148953:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.148971:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760793.149704:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99500 - object 0x280000401:2 offset 80973824<>80977920 00000100:00100000:1.0:1789760793.149770:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503920640:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760793.161199:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987076 gen 2 00000008:00010000:1.0:1789760793.161261:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760793.161270:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.161283:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.161423:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760793.161459:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503920640:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760793.162293:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19769, 19769] 00000080:00200000:2.0:1789760793.162360:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.162366:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.162383:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80977920 i_blocks 158184 00000020:00200000:2.0:1789760793.162388:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.162400:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.162435:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80977920, count: 4096 00000080:00200000:2.0:1789760793.162443:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80977920, bytes: 4096 00000080:00200000:2.0:1789760793.162455:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.162465:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760793.162467:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80977920, 80982016) 0 0 00000080:00200000:2.0:1789760793.162472:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.162488:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19770, 19770] 00020000:00200000:2.0:1789760793.162496:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80977920 chunk: [80977920, 80982016) 80982016, 4096 00020000:00200000:2.0:1789760793.162501:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.162597:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19770, 19770] 00000008:00010000:2.0:1789760793.162626:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.162633:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760793.162646:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80977920, 80982016) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.162661:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80977920=0x4d3a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.162727:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.162730:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80982016 00000020:00010000:2.0:1789760793.162738:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.162742:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.162745:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.162748:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.162750:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.162759:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.162765:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.162790:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760793.162891:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80977920<>80982016 00000100:00100000:3.0:1789760793.162977:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503920768:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760793.175546:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987077 gen 2 00000008:00010000:2.0:1789760793.175609:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760793.175618:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.175629:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.175656:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760793.175688:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503920768:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.175767:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19770, 19770] 00000080:00200000:1.0:1789760793.175785:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.175791:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.175807:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80982016 i_blocks 158192 00000020:00200000:1.0:1789760793.175811:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.175821:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.175859:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80982016, count: 4096 00000080:00200000:1.0:1789760793.175867:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80982016, bytes: 4096 00000080:00200000:1.0:1789760793.175878:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.175886:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.175888:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80982016, 80986112) 0 0 00000080:00200000:1.0:1789760793.175893:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.176458:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19771, 19771] 00020000:00200000:1.0:1789760793.176468:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80982016 chunk: [80982016, 80986112) 80986112, 4096 00020000:00200000:1.0:1789760793.176474:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.176496:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19771, 19771] 00000008:00010000:1.0:1789760793.176515:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.176522:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.177284:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80982016, 80986112) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.177413:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80982016=0x4d3b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.177481:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.177485:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80986112 00000020:00010000:1.0:1789760793.177493:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.177496:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.177500:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.177503:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.177506:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.177518:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.177525:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.177550:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760793.177814:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 80982016<>80986112 00000100:00100000:0.0:1789760793.177893:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503920896:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.185915:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987078 gen 2 00000008:00010000:0.0:1789760793.185964:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:0.0:1789760793.185973:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.185984:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.186007:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000080:00200000:1.0:1789760793.186149:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19771, 19771] 00000080:00200000:1.0:1789760793.186167:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.186266:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.186284:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80986112 i_blocks 158200 00000020:00200000:1.0:1789760793.186288:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.186299:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.186437:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80986112, count: 4096 00000080:00200000:1.0:1789760793.186447:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80986112, bytes: 4096 00000080:00200000:1.0:1789760793.186457:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.186466:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.186468:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80986112, 80990208) 0 0 00000080:00200000:1.0:1789760793.186473:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000100:00100000:0.0:1789760793.186501:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503920896:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.186558:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19772, 19772] 00020000:00200000:1.0:1789760793.186568:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80986112 chunk: [80986112, 80990208) 80990208, 4096 00020000:00200000:1.0:1789760793.186574:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.186604:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19772, 19772] 00000008:00010000:1.0:1789760793.186621:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.186626:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.186636:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80986112, 80990208) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.186649:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80986112=0x4d3c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.186701:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.186703:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80990208 00000020:00010000:1.0:1789760793.186712:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.186715:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.186718:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.186722:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.186725:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.186732:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.186739:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.186755:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760793.186846:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 80986112<>80990208 00000100:00100000:0.0:1789760793.186912:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503921024:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.197777:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987079 gen 2 00000008:00010000:0.0:1789760793.197830:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d583c0 00000020:00200000:0.0:1789760793.197839:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.197851:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.197876:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d583c0 00000100:00100000:0.0:1789760793.199296:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503921024:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.200013:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19772, 19772] 00000080:00200000:1.0:1789760793.200052:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.200058:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.200076:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80990208 i_blocks 158208 00000020:00200000:1.0:1789760793.200082:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.200095:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.200129:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80990208, count: 4096 00000080:00200000:1.0:1789760793.200149:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80990208, bytes: 4096 00000080:00200000:1.0:1789760793.200163:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.200173:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.200175:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80990208, 80994304) 0 0 00000080:00200000:1.0:1789760793.200180:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.200198:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19773, 19773] 00020000:00200000:1.0:1789760793.200206:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80990208 chunk: [80990208, 80994304) 80994304, 4096 00020000:00200000:1.0:1789760793.200211:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.200234:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19773, 19773] 00000008:00010000:1.0:1789760793.200253:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.200260:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.200271:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80990208, 80994304) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.200286:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80990208=0x4d3d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.200354:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.200357:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80994304 00000020:00010000:1.0:1789760793.200365:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.200369:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.200372:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.200375:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.200378:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.200386:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.200394:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.200413:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760793.200532:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 80990208<>80994304 00000100:00100000:0.0:1789760793.201013:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760793.201055:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503921152:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.210502:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987080 gen 2 00000008:00010000:0.0:1789760793.210551:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58540 00000020:00200000:0.0:1789760793.210561:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.210633:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.210753:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58540 00000100:00100000:0.0:1789760793.210785:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503921152:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.210875:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19773, 19773] 00000080:00200000:1.0:1789760793.210892:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.210951:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.211054:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80994304 i_blocks 158216 00000020:00200000:1.0:1789760793.211059:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.211070:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.211105:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80994304, count: 4096 00000080:00200000:1.0:1789760793.211114:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80994304, bytes: 4096 00000080:00200000:1.0:1789760793.211127:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.211205:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.211208:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80994304, 80998400) 0 0 00000080:00200000:1.0:1789760793.211213:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.211230:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19774, 19774] 00020000:00200000:1.0:1789760793.211239:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80994304 chunk: [80994304, 80998400) 80998400, 4096 00020000:00200000:1.0:1789760793.211243:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.211263:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19774, 19774] 00000008:00010000:1.0:1789760793.211280:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.211288:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.211299:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80994304, 80998400) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.211312:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80994304=0x4d3e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.211366:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.211369:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 80998400 00000020:00010000:1.0:1789760793.211377:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.211381:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.211384:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.211388:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.211390:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.211399:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.211405:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.211423:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760793.211521:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 80994304<>80998400 00000100:00100000:0.0:1789760793.211598:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503921280:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.233258:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987081 gen 2 00000008:00010000:0.0:1789760793.233329:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58a80 00000020:00200000:0.0:1789760793.233338:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.233349:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.233374:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58a80 00000100:00100000:0.0:1789760793.233406:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503921280:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760793.233698:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19774, 19774] 00000080:00200000:2.0:1789760793.233715:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.233722:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.233740:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 80998400 i_blocks 158224 00000020:00200000:2.0:1789760793.233744:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.233754:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.233793:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 80998400, count: 4096 00000080:00200000:2.0:1789760793.233801:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 80998400, bytes: 4096 00000080:00200000:2.0:1789760793.233813:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.233822:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760793.233824:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [80998400, 81002496) 0 0 00000080:00200000:2.0:1789760793.233829:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.233846:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19775, 19775] 00020000:00200000:2.0:1789760793.233855:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 80998400 chunk: [80998400, 81002496) 81002496, 4096 00020000:00200000:2.0:1789760793.233859:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.233879:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19775, 19775] 00000008:00010000:2.0:1789760793.233898:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.233904:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760793.233916:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [80998400, 81002496) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.233930:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=80998400=0x4d3f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.233988:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.233991:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81002496 00000020:00010000:2.0:1789760793.233999:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.234003:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.234006:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.234009:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.234013:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.234022:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.234028:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.234091:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760793.234203:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 80998400<>81002496 00000100:00100000:2.0:1789760793.234252:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760793.234272:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503921408:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760793.244759:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987082 gen 2 00000008:00010000:2.0:1789760793.244808:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760793.244816:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.244828:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.244914:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760793.244947:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503921408:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760793.246395:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19775, 19775] 00000080:00200000:0.0:1789760793.246475:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.246483:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.246502:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81002496 i_blocks 158232 00000020:00200000:0.0:1789760793.246508:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.246527:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.246564:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81002496, count: 4096 00000080:00200000:0.0:1789760793.246573:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81002496, bytes: 4096 00000080:00200000:0.0:1789760793.246597:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.246608:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760793.246611:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81002496, 81006592) 0 0 00000080:00200000:0.0:1789760793.246615:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.246633:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19776, 19776] 00020000:00200000:0.0:1789760793.246642:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81002496 chunk: [81002496, 81006592) 81006592, 4096 00020000:00200000:0.0:1789760793.246647:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.247448:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19776, 19776] 00000008:00010000:0.0:1789760793.247479:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.247486:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760793.247811:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81002496, 81006592) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.247829:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81002496=0x4d40000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.247910:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.247914:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81006592 00000020:00010000:0.0:1789760793.247923:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.247926:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.247929:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.247933:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.247936:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.247947:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.247956:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.247988:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760793.248264:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99500 - object 0x280000401:2 offset 81002496<>81006592 00000100:00100000:1.0:1789760793.248344:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503921536:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760793.261396:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987083 gen 2 00000008:00010000:1.0:1789760793.261449:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760793.261457:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.261469:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.261513:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760793.261548:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503921536:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760793.261868:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19776, 19776] 00000080:00200000:2.0:1789760793.261883:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.261889:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.261904:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81006592 i_blocks 158240 00000020:00200000:2.0:1789760793.261908:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.261917:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.261952:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81006592, count: 4096 00000080:00200000:2.0:1789760793.261958:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81006592, bytes: 4096 00000080:00200000:2.0:1789760793.261969:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.261977:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760793.261979:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81006592, 81010688) 0 0 00000080:00200000:2.0:1789760793.261983:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.261998:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19777, 19777] 00020000:00200000:2.0:1789760793.262006:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81006592 chunk: [81006592, 81010688) 81010688, 4096 00020000:00200000:2.0:1789760793.262012:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.262030:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19777, 19777] 00000008:00010000:2.0:1789760793.262077:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.262084:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760793.262095:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81006592, 81010688) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.262107:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81006592=0x4d41000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.262160:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.262163:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81010688 00000020:00010000:2.0:1789760793.262171:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.262174:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.262177:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.262181:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.262183:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.262191:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.262197:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.262212:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760793.262304:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 81006592<>81010688 00000100:00100000:3.0:1789760793.262463:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760793.262482:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503921664:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760793.274959:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987084 gen 2 00000008:00010000:2.0:1789760793.275014:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760793.275023:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.275068:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.275204:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760793.275237:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503921664:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.276241:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19777, 19777] 00000080:00200000:1.0:1789760793.276260:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.276266:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.276282:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81010688 i_blocks 158248 00000020:00200000:1.0:1789760793.276287:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.276298:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.276449:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81010688, count: 4096 00000080:00200000:1.0:1789760793.276458:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81010688, bytes: 4096 00000080:00200000:1.0:1789760793.276471:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.276480:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.276483:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81010688, 81014784) 0 0 00000080:00200000:1.0:1789760793.276488:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.276503:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19778, 19778] 00020000:00200000:1.0:1789760793.276511:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81010688 chunk: [81010688, 81014784) 81014784, 4096 00020000:00200000:1.0:1789760793.276516:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.276534:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19778, 19778] 00000008:00010000:1.0:1789760793.276550:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.276556:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.276567:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81010688, 81014784) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.276590:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81010688=0x4d42000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.276767:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.276771:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81014784 00000020:00010000:1.0:1789760793.276780:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.276784:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.276786:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.276790:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.276792:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.276802:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.276808:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.276833:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760793.276940:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81010688<>81014784 00000100:00100000:0.0:1789760793.277067:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503921792:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.295436:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987085 gen 2 00000008:00010000:0.0:1789760793.295485:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d586c0 00000020:00200000:0.0:1789760793.295493:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.295505:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.295530:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d586c0 00000100:00100000:0.0:1789760793.295560:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503921792:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.295702:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19778, 19778] 00000080:00200000:1.0:1789760793.295719:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.295725:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.295741:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81014784 i_blocks 158256 00000020:00200000:1.0:1789760793.295745:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.295755:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.295789:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81014784, count: 4096 00000080:00200000:1.0:1789760793.295796:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81014784, bytes: 4096 00000080:00200000:1.0:1789760793.295806:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.295815:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.295818:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81014784, 81018880) 0 0 00000080:00200000:1.0:1789760793.295822:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.295838:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19779, 19779] 00020000:00200000:1.0:1789760793.295846:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81014784 chunk: [81014784, 81018880) 81018880, 4096 00020000:00200000:1.0:1789760793.295851:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.295869:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19779, 19779] 00000008:00010000:1.0:1789760793.295886:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.295892:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.295904:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81014784, 81018880) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.295917:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81014784=0x4d43000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.295971:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.295973:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81018880 00000020:00010000:1.0:1789760793.295981:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.295984:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.295988:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.295991:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.295994:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.296001:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.296007:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.296022:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760793.296352:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81014784<>81018880 00000100:00100000:1.0:1789760793.296393:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503921920:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760793.311183:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987086 gen 2 00000008:00010000:1.0:1789760793.311234:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:1.0:1789760793.311243:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.311255:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.311281:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:1.0:1789760793.311314:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503921920:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760793.311426:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19779, 19779] 00000080:00200000:2.0:1789760793.311442:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.311448:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.311463:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81018880 i_blocks 158264 00000020:00200000:2.0:1789760793.311466:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.311477:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.311513:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81018880, count: 4096 00000080:00200000:2.0:1789760793.311520:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81018880, bytes: 4096 00000080:00200000:2.0:1789760793.311531:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.311540:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760793.311542:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81018880, 81022976) 0 0 00000080:00200000:2.0:1789760793.311547:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.311561:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19780, 19780] 00020000:00200000:2.0:1789760793.311570:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81018880 chunk: [81018880, 81022976) 81022976, 4096 00020000:00200000:2.0:1789760793.311575:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.311606:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19780, 19780] 00000008:00010000:2.0:1789760793.311634:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.311640:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760793.311650:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81018880, 81022976) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.311664:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81018880=0x4d44000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.311722:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.311725:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81022976 00000020:00010000:2.0:1789760793.311733:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.311736:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.311739:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.311743:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.311746:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.311755:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.311761:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.311779:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760793.311881:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 81018880<>81022976 00000100:00100000:3.0:1789760793.312151:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503922048:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760793.330063:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987087 gen 2 00000008:00010000:2.0:1789760793.330116:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760793.330124:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.330135:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.330245:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760793.330276:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503922048:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.330472:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19780, 19780] 00000080:00200000:1.0:1789760793.330678:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.330686:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.330703:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81022976 i_blocks 158272 00000020:00200000:1.0:1789760793.330707:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.330718:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.330754:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81022976, count: 4096 00000080:00200000:1.0:1789760793.330761:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81022976, bytes: 4096 00000080:00200000:1.0:1789760793.330775:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.330784:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.330786:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81022976, 81027072) 0 0 00000080:00200000:1.0:1789760793.330789:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.330806:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19781, 19781] 00020000:00200000:1.0:1789760793.330816:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81022976 chunk: [81022976, 81027072) 81027072, 4096 00020000:00200000:1.0:1789760793.330820:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.330839:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19781, 19781] 00000008:00010000:1.0:1789760793.330857:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.330863:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.330874:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81022976, 81027072) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.330887:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81022976=0x4d45000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.330946:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.330950:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81027072 00000020:00010000:1.0:1789760793.330957:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.330961:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.330964:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.330967:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.330984:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.330991:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.330998:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.331015:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760793.331160:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81022976<>81027072 00000100:00100000:1.0:1789760793.331196:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760793.331215:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503922176:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760793.347548:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987088 gen 2 00000008:00010000:1.0:1789760793.347607:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:1.0:1789760793.347615:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.347627:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.347691:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:1.0:1789760793.347721:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503922176:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760793.347799:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19781, 19781] 00000080:00200000:2.0:1789760793.347884:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.347891:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.348103:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81027072 i_blocks 158280 00000020:00200000:2.0:1789760793.348108:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.348119:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.348155:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81027072, count: 4096 00000080:00200000:2.0:1789760793.348162:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81027072, bytes: 4096 00000080:00200000:2.0:1789760793.348177:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.348187:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760793.348189:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81027072, 81031168) 0 0 00000080:00200000:2.0:1789760793.348193:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.348210:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19782, 19782] 00020000:00200000:2.0:1789760793.348220:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81027072 chunk: [81027072, 81031168) 81031168, 4096 00020000:00200000:2.0:1789760793.348225:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.348304:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19782, 19782] 00000008:00010000:2.0:1789760793.348325:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.348332:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760793.348345:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81027072, 81031168) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.348427:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81027072=0x4d46000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.348494:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.348496:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81031168 00000020:00010000:2.0:1789760793.348506:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.348510:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.348513:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.348516:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.348519:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.348528:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.348535:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.348569:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760793.348683:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 81027072<>81031168 00000100:00100000:3.0:1789760793.348800:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503922304:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760793.376100:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987089 gen 2 00000008:00010000:2.0:1789760793.376147:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:2.0:1789760793.376155:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.376166:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.376240:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:2.0:1789760793.376270:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503922304:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.379947:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19782, 19782] 00000080:00200000:1.0:1789760793.379975:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.379982:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.380002:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81031168 i_blocks 158288 00000020:00200000:1.0:1789760793.380007:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.380026:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.380078:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81031168, count: 4096 00000080:00200000:1.0:1789760793.380087:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81031168, bytes: 4096 00000080:00200000:1.0:1789760793.380365:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.380376:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.380378:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81031168, 81035264) 0 0 00000080:00200000:1.0:1789760793.380384:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.380400:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19783, 19783] 00020000:00200000:1.0:1789760793.380410:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81031168 chunk: [81031168, 81035264) 81035264, 4096 00020000:00200000:1.0:1789760793.380414:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.380437:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19783, 19783] 00000008:00010000:1.0:1789760793.380458:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.380463:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.380476:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81031168, 81035264) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.380490:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81031168=0x4d47000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.380565:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.380568:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81035264 00000020:00010000:1.0:1789760793.380761:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.380765:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.380825:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.380829:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.380833:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.380843:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.380849:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.380878:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760793.380985:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81031168<>81035264 00000100:00100000:0.0:1789760793.382839:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503922432:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.403185:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987090 gen 2 00000008:00010000:0.0:1789760793.403237:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:0.0:1789760793.403246:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.403257:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.403283:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:0.0:1789760793.403315:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503922432:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.403322:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19783, 19783] 00000080:00200000:1.0:1789760793.403341:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.403347:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.403362:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81035264 i_blocks 158296 00000020:00200000:1.0:1789760793.403366:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.403377:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.403411:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81035264, count: 4096 00000080:00200000:1.0:1789760793.403419:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81035264, bytes: 4096 00000080:00200000:1.0:1789760793.403429:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.403439:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.403441:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81035264, 81039360) 0 0 00000080:00200000:1.0:1789760793.403447:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.403464:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19784, 19784] 00020000:00200000:1.0:1789760793.403472:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81035264 chunk: [81035264, 81039360) 81039360, 4096 00020000:00200000:1.0:1789760793.403477:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.403495:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19784, 19784] 00000008:00010000:1.0:1789760793.403512:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.403519:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.403530:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81035264, 81039360) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.403544:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81035264=0x4d48000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.403624:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.403627:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81039360 00000020:00010000:1.0:1789760793.403636:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.403639:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.403642:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.403646:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.403648:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.403657:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.403664:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.403680:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760793.403787:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 81035264<>81039360 00000100:00100000:0.0:1789760793.406827:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760793.406872:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503922560:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.423865:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987091 gen 2 00000008:00010000:0.0:1789760793.423916:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58000 00000020:00200000:0.0:1789760793.423923:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.423935:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.423962:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58000 00000100:00100000:0.0:1789760793.424009:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503922560:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.424269:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19784, 19784] 00000080:00200000:1.0:1789760793.424288:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.424295:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.424311:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81039360 i_blocks 158304 00000020:00200000:1.0:1789760793.424315:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.424324:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.424362:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81039360, count: 4096 00000080:00200000:1.0:1789760793.424369:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81039360, bytes: 4096 00000080:00200000:1.0:1789760793.424380:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.424390:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.424392:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81039360, 81043456) 0 0 00000080:00200000:1.0:1789760793.424398:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.424416:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19785, 19785] 00020000:00200000:1.0:1789760793.424424:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81039360 chunk: [81039360, 81043456) 81043456, 4096 00020000:00200000:1.0:1789760793.424429:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.424449:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19785, 19785] 00000008:00010000:1.0:1789760793.424468:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.424475:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.424488:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81039360, 81043456) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.424503:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81039360=0x4d49000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.424715:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.424719:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81043456 00000020:00010000:1.0:1789760793.424728:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.424731:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.424735:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.424739:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.424741:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.424750:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.424759:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.424778:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760793.424893:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 81039360<>81043456 00000100:00100000:0.0:1789760793.425021:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503922688:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.440966:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987092 gen 2 00000008:00010000:0.0:1789760793.441077:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:0.0:1789760793.441087:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.441099:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.441120:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000080:00200000:1.0:1789760793.441148:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19785, 19785] 00000100:00100000:0.0:1789760793.441154:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503922688:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.441169:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.441176:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.441193:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81043456 i_blocks 158312 00000020:00200000:1.0:1789760793.441197:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.441209:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.441248:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81043456, count: 4096 00000080:00200000:1.0:1789760793.441256:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81043456, bytes: 4096 00000080:00200000:1.0:1789760793.441267:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.441276:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.441279:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81043456, 81047552) 0 0 00000080:00200000:1.0:1789760793.441284:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.441300:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19786, 19786] 00020000:00200000:1.0:1789760793.441311:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81043456 chunk: [81043456, 81047552) 81047552, 4096 00020000:00200000:1.0:1789760793.441316:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.441335:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19786, 19786] 00000008:00010000:1.0:1789760793.441354:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.441360:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.441372:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81043456, 81047552) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.441386:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81043456=0x4d4a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.441446:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.441449:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81047552 00000020:00010000:1.0:1789760793.441456:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.441459:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.441462:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.441465:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.441469:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.441476:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.441482:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.441497:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58780 00000008:00100000:1.0:1789760793.443496:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 81043456<>81047552 00000100:00100000:0.0:1789760793.443689:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760793.443715:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503922816:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.461280:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987093 gen 2 00000008:00010000:0.0:1789760793.461326:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58780 00000020:00200000:0.0:1789760793.461334:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.461342:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.461364:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58780 00000100:00100000:0.0:1789760793.461388:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503922816:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.461967:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19786, 19786] 00000080:00200000:1.0:1789760793.461979:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.461984:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.461997:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81047552 i_blocks 158320 00000020:00200000:1.0:1789760793.462001:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.462008:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.462069:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81047552, count: 4096 00000080:00200000:1.0:1789760793.462076:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81047552, bytes: 4096 00000080:00200000:1.0:1789760793.462085:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.462092:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.462094:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81047552, 81051648) 0 0 00000080:00200000:1.0:1789760793.462099:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.462111:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19787, 19787] 00020000:00200000:1.0:1789760793.462118:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81047552 chunk: [81047552, 81051648) 81051648, 4096 00020000:00200000:1.0:1789760793.462122:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.462137:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19787, 19787] 00000008:00010000:1.0:1789760793.462151:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.462157:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.462167:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81047552, 81051648) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.462179:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81047552=0x4d4b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.462224:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.462227:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81051648 00000020:00010000:1.0:1789760793.462234:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.462237:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.462240:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.462242:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.462245:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.462251:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.462256:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.462270:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760793.462359:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 81047552<>81051648 00000100:00100000:0.0:1789760793.462440:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760793.462456:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503922944:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.479739:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987094 gen 2 00000008:00010000:0.0:1789760793.479777:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:0.0:1789760793.479783:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.479793:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.479818:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:0.0:1789760793.479843:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503922944:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.479944:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19787, 19787] 00000080:00200000:1.0:1789760793.479957:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.479962:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.479973:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81051648 i_blocks 158328 00000020:00200000:1.0:1789760793.479977:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.479985:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.480008:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81051648, count: 4096 00000080:00200000:1.0:1789760793.480014:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81051648, bytes: 4096 00000080:00200000:1.0:1789760793.480024:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.480030:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.480051:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81051648, 81055744) 0 0 00000080:00200000:1.0:1789760793.480055:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.480068:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19788, 19788] 00020000:00200000:1.0:1789760793.480075:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81051648 chunk: [81051648, 81055744) 81055744, 4096 00020000:00200000:1.0:1789760793.480079:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.480093:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19788, 19788] 00000008:00010000:1.0:1789760793.480106:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.480110:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.480118:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81051648, 81055744) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.480128:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81051648=0x4d4c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.480166:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.480169:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81055744 00000020:00010000:1.0:1789760793.480175:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.480178:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.480181:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.480184:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.480187:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.480192:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.480196:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.480208:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760793.480278:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 81051648<>81055744 00000100:00100000:0.0:1789760793.480336:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503923072:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.500819:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987095 gen 2 00000008:00010000:0.0:1789760793.500855:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760793.500861:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.500869:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.500890:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760793.500913:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503923072:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.501208:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19788, 19788] 00000080:00200000:1.0:1789760793.501221:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.501226:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.501239:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81055744 i_blocks 158336 00000020:00200000:1.0:1789760793.501242:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.501250:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.501276:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81055744, count: 4096 00000080:00200000:1.0:1789760793.501282:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81055744, bytes: 4096 00000080:00200000:1.0:1789760793.501289:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.501296:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.501298:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81055744, 81059840) 0 0 00000080:00200000:1.0:1789760793.501302:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.501314:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19789, 19789] 00020000:00200000:1.0:1789760793.501321:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81055744 chunk: [81055744, 81059840) 81059840, 4096 00020000:00200000:1.0:1789760793.501325:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.501409:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19789, 19789] 00000008:00010000:1.0:1789760793.501424:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.501429:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.501438:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81055744, 81059840) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.501448:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81055744=0x4d4d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.501493:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.501496:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81059840 00000020:00010000:1.0:1789760793.501502:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.501506:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.501508:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.501511:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.501514:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.501520:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.501525:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.501539:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760793.501926:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 81055744<>81059840 00000100:00100000:0.0:1789760793.502073:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503923200:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.530903:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987096 gen 2 00000008:00010000:0.0:1789760793.530940:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760793.530947:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.530955:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.530977:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:0.0:1789760793.531001:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503923200:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.531212:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19789, 19789] 00000080:00200000:1.0:1789760793.531225:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.531230:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.531241:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81059840 i_blocks 158344 00000020:00200000:1.0:1789760793.531245:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.531252:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.531277:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81059840, count: 4096 00000080:00200000:1.0:1789760793.531282:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81059840, bytes: 4096 00000080:00200000:1.0:1789760793.531290:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.531308:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.531311:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81059840, 81063936) 0 0 00000080:00200000:1.0:1789760793.531315:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.531328:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19790, 19790] 00020000:00200000:1.0:1789760793.531335:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81059840 chunk: [81059840, 81063936) 81063936, 4096 00020000:00200000:1.0:1789760793.531339:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.531354:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19790, 19790] 00000008:00010000:1.0:1789760793.531367:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.531498:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.531923:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81059840, 81063936) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.531935:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81059840=0x4d4e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.531983:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.531986:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81063936 00000020:00010000:1.0:1789760793.531992:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.531995:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.531998:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.532001:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.532004:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.532010:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.532015:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.532032:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760793.532144:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81059840<>81063936 00000100:00100000:0.0:1789760793.532510:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760793.532526:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503923328:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.551988:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987097 gen 2 00000008:00010000:0.0:1789760793.552021:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760793.552028:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.552062:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.552084:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000100:00100000:0.0:1789760793.552109:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503923328:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.552204:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19790, 19790] 00000080:00200000:1.0:1789760793.552217:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.552222:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.552234:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81063936 i_blocks 158352 00000020:00200000:1.0:1789760793.552237:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.552245:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.552270:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81063936, count: 4096 00000080:00200000:1.0:1789760793.552276:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81063936, bytes: 4096 00000080:00200000:1.0:1789760793.552283:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.552290:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.552291:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81063936, 81068032) 0 0 00000080:00200000:1.0:1789760793.552296:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.552320:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19791, 19791] 00020000:00200000:1.0:1789760793.552327:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81063936 chunk: [81063936, 81068032) 81068032, 4096 00020000:00200000:1.0:1789760793.552331:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.552345:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19791, 19791] 00000008:00010000:1.0:1789760793.552358:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.552364:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.552372:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81063936, 81068032) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.552382:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81063936=0x4d4f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.552428:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.552431:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81068032 00000020:00010000:1.0:1789760793.552439:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.552443:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.552445:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.552448:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.552451:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.552457:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.552463:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.552476:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760793.552715:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81063936<>81068032 00000100:00100000:0.0:1789760793.553115:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760793.553129:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503923456:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.572486:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987098 gen 2 00000008:00010000:0.0:1789760793.572521:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760793.572528:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.572536:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.572555:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000080:00200000:1.0:1789760793.572623:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19791, 19791] 00000080:00200000:1.0:1789760793.572637:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.572642:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.572654:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81068032 i_blocks 158360 00000020:00200000:1.0:1789760793.572657:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.572665:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.572691:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81068032, count: 4096 00000080:00200000:1.0:1789760793.572698:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81068032, bytes: 4096 00000080:00200000:1.0:1789760793.572706:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.572713:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.572715:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81068032, 81072128) 0 0 00000080:00200000:1.0:1789760793.572719:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.572732:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19792, 19792] 00020000:00200000:1.0:1789760793.572739:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81068032 chunk: [81068032, 81072128) 81072128, 4096 00020000:00200000:1.0:1789760793.572744:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.572758:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19792, 19792] 00000008:00010000:1.0:1789760793.572771:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.572776:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.572785:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81068032, 81072128) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.572795:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81068032=0x4d50000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.572830:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.572833:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81072128 00000020:00010000:1.0:1789760793.572840:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.572843:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.572846:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.572849:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.572852:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.572858:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.572863:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1789760793.573526:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503923456:192.168.203.158@tcp:4:dd.0 00000008:00010000:0.0:1789760793.573568:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:0.0:1789760793.573682:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a300 - object 0x280000401:2 offset 81068032<>81072128 00000100:00100000:0.0:1789760793.573719:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503923584:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.584437:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987099 gen 2 00000008:00010000:0.0:1789760793.584473:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760793.584480:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.584488:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.584507:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:0.0:1789760793.584531:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503923584:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.584565:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19792, 19792] 00000080:00200000:1.0:1789760793.584588:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.584594:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.584606:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81072128 i_blocks 158368 00000020:00200000:1.0:1789760793.584609:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.584617:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.584641:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81072128, count: 4096 00000080:00200000:1.0:1789760793.584648:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81072128, bytes: 4096 00000080:00200000:1.0:1789760793.584656:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.584663:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.584666:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81072128, 81076224) 0 0 00000080:00200000:1.0:1789760793.584669:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.584682:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19793, 19793] 00020000:00200000:1.0:1789760793.584689:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81072128 chunk: [81072128, 81076224) 81076224, 4096 00020000:00200000:1.0:1789760793.584693:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.584707:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19793, 19793] 00000008:00010000:1.0:1789760793.584720:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.584725:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.584733:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81072128, 81076224) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.584744:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81072128=0x4d51000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.584786:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.584789:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81076224 00000020:00010000:1.0:1789760793.584795:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.584798:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.584801:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.584804:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.584807:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.584812:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.584817:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.584830:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760793.584907:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 81072128<>81076224 00000100:00100000:0.0:1789760793.584999:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503923712:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.601935:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987100 gen 2 00000008:00010000:0.0:1789760793.601972:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760793.601978:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.601988:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.602010:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760793.602077:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503923712:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760793.602162:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19793, 19793] 00000080:00200000:2.0:1789760793.602175:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.602180:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.602192:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81076224 i_blocks 158376 00000020:00200000:2.0:1789760793.602196:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.602204:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.602228:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81076224, count: 4096 00000080:00200000:2.0:1789760793.602233:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81076224, bytes: 4096 00000080:00200000:2.0:1789760793.602244:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.602251:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760793.602253:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81076224, 81080320) 0 0 00000080:00200000:2.0:1789760793.602256:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.602269:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19794, 19794] 00020000:00200000:2.0:1789760793.602276:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81076224 chunk: [81076224, 81080320) 81080320, 4096 00020000:00200000:2.0:1789760793.602280:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.602294:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19794, 19794] 00000008:00010000:2.0:1789760793.603585:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.603593:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760793.603610:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81076224, 81080320) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.603624:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81076224=0x4d52000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.603677:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.603680:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81080320 00000020:00010000:2.0:1789760793.603686:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.603688:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.603692:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.603694:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.603697:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.603706:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.603713:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.603732:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760793.603813:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 81076224<>81080320 00000100:00100000:3.0:1789760793.603994:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503923840:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760793.616934:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987101 gen 2 00000008:00010000:3.0:1789760793.616971:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760793.616978:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.616987:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.617007:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760793.617031:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503923840:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760793.618429:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19794, 19794] 00000080:00200000:2.0:1789760793.618447:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.618454:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.618468:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81080320 i_blocks 158384 00000020:00200000:2.0:1789760793.618471:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.618482:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.618507:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81080320, count: 4096 00000080:00200000:2.0:1789760793.618513:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81080320, bytes: 4096 00000080:00200000:2.0:1789760793.618523:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.618530:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760793.618533:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81080320, 81084416) 0 0 00000080:00200000:2.0:1789760793.618536:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.618548:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19795, 19795] 00020000:00200000:2.0:1789760793.618555:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81080320 chunk: [81080320, 81084416) 81084416, 4096 00020000:00200000:2.0:1789760793.618559:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.618584:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19795, 19795] 00000008:00010000:2.0:1789760793.618599:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.618603:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760793.618613:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81080320, 81084416) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.618625:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81080320=0x4d53000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.618680:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.618683:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81084416 00000020:00010000:2.0:1789760793.618690:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.618693:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.618695:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.618698:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.618701:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.618708:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.618714:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.618729:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ce40 00000008:00100000:2.0:1789760793.618807:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24e00 - object 0x280000401:2 offset 81080320<>81084416 00000100:00100000:3.0:1789760793.619245:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760793.619263:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503923968:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760793.635641:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987102 gen 2 00000008:00010000:3.0:1789760793.635686:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760793.635694:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.635703:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.635720:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760793.635745:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503923968:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760793.636015:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19795, 19795] 00000080:00200000:2.0:1789760793.636028:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.636077:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.636089:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81084416 i_blocks 158392 00000020:00200000:2.0:1789760793.636093:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.636101:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.636128:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81084416, count: 4096 00000080:00200000:2.0:1789760793.636133:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81084416, bytes: 4096 00000080:00200000:2.0:1789760793.636141:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.636148:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760793.636150:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81084416, 81088512) 0 0 00000080:00200000:2.0:1789760793.636154:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.636166:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19796, 19796] 00020000:00200000:2.0:1789760793.636173:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81084416 chunk: [81084416, 81088512) 81088512, 4096 00020000:00200000:2.0:1789760793.636178:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.636191:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19796, 19796] 00000008:00010000:2.0:1789760793.636204:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.636208:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760793.636217:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81084416, 81088512) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.636227:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81084416=0x4d54000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.636271:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.636274:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81088512 00000020:00010000:2.0:1789760793.636280:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.636283:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.636286:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.636289:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.636292:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.636299:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.636305:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.636319:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c000 00000008:00100000:2.0:1789760793.636809:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27480 - object 0x280000401:2 offset 81084416<>81088512 00000100:00100000:3.0:1789760793.636889:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503924096:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760793.653161:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987103 gen 2 00000008:00010000:3.0:1789760793.653194:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760793.653201:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.653209:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.653229:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760793.653253:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503924096:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760793.653522:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19796, 19796] 00000080:00200000:2.0:1789760793.653535:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.653540:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.653555:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81088512 i_blocks 158400 00000020:00200000:2.0:1789760793.653559:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.653566:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.653598:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81088512, count: 4096 00000080:00200000:2.0:1789760793.653605:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81088512, bytes: 4096 00000080:00200000:2.0:1789760793.653613:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.653743:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760793.653745:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81088512, 81092608) 0 0 00000080:00200000:2.0:1789760793.653749:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.653762:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19797, 19797] 00020000:00200000:2.0:1789760793.653768:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81088512 chunk: [81088512, 81092608) 81092608, 4096 00020000:00200000:2.0:1789760793.653772:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.653788:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19797, 19797] 00000008:00010000:2.0:1789760793.653802:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.653816:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760793.653825:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81088512, 81092608) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.653837:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81088512=0x4d55000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.653895:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.653898:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81092608 00000020:00010000:2.0:1789760793.653904:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.653907:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.653910:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.653913:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.653916:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.653923:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.653928:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.653943:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cf00 00000008:00100000:2.0:1789760793.654016:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26680 - object 0x280000401:2 offset 81088512<>81092608 00000100:00100000:3.0:1789760793.654624:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503924224:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760793.676010:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987104 gen 2 00000008:00010000:3.0:1789760793.676064:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760793.676071:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.676079:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.676104:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760793.676140:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503924224:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.676185:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19797, 19797] 00000080:00200000:1.0:1789760793.676198:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.676204:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.676216:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81092608 i_blocks 158408 00000020:00200000:1.0:1789760793.676219:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.676228:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.676251:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81092608, count: 4096 00000080:00200000:1.0:1789760793.676257:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81092608, bytes: 4096 00000080:00200000:1.0:1789760793.676266:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.676273:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.676275:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81092608, 81096704) 0 0 00000080:00200000:1.0:1789760793.676279:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.676291:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19798, 19798] 00020000:00200000:1.0:1789760793.676299:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81092608 chunk: [81092608, 81096704) 81096704, 4096 00020000:00200000:1.0:1789760793.676303:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.676316:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19798, 19798] 00000008:00010000:1.0:1789760793.676328:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.676333:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.676342:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81092608, 81096704) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.676352:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81092608=0x4d56000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.676400:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.676403:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81096704 00000020:00010000:1.0:1789760793.676411:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.676414:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.676417:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.676421:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.676424:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.676430:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.676436:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.676450:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760793.676522:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 81092608<>81096704 00000100:00100000:0.0:1789760793.676609:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503924352:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.689147:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987105 gen 2 00000008:00010000:0.0:1789760793.689182:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:0.0:1789760793.689188:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.689197:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.689221:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:0.0:1789760793.689245:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503924352:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.689392:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19798, 19798] 00000080:00200000:1.0:1789760793.689413:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.689419:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.689430:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81096704 i_blocks 158416 00000020:00200000:1.0:1789760793.689434:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.689441:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.689467:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81096704, count: 4096 00000080:00200000:1.0:1789760793.689472:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81096704, bytes: 4096 00000080:00200000:1.0:1789760793.689490:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.689499:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.689501:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81096704, 81100800) 0 0 00000080:00200000:1.0:1789760793.689505:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.689517:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19799, 19799] 00020000:00200000:1.0:1789760793.689524:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81096704 chunk: [81096704, 81100800) 81100800, 4096 00020000:00200000:1.0:1789760793.689527:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.689542:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19799, 19799] 00000008:00010000:1.0:1789760793.689554:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.689559:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.689567:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81096704, 81100800) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.689725:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81096704=0x4d57000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.689775:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.689778:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81100800 00000020:00010000:1.0:1789760793.689784:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.689788:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.689791:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.689794:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.689796:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.689803:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.689808:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.689822:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58780 00000008:00100000:1.0:1789760793.689899:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 81096704<>81100800 00000100:00100000:0.0:1789760793.690014:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503924480:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.700127:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987106 gen 2 00000008:00010000:0.0:1789760793.700163:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58780 00000020:00200000:0.0:1789760793.700170:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.700178:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.700234:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58780 00000100:00100000:0.0:1789760793.700258:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503924480:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.700912:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19799, 19799] 00000080:00200000:1.0:1789760793.701031:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.701059:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.701072:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81100800 i_blocks 158424 00000020:00200000:1.0:1789760793.701076:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.701084:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.701107:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81100800, count: 4096 00000080:00200000:1.0:1789760793.701112:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81100800, bytes: 4096 00000080:00200000:1.0:1789760793.701119:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.701124:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.701126:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81100800, 81104896) 0 0 00000080:00200000:1.0:1789760793.701129:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.701140:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19800, 19800] 00020000:00200000:1.0:1789760793.701145:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81100800 chunk: [81100800, 81104896) 81104896, 4096 00020000:00200000:1.0:1789760793.701148:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.701160:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19800, 19800] 00000008:00010000:1.0:1789760793.701170:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.701174:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.701182:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81100800, 81104896) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.701190:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81100800=0x4d58000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.701228:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.701230:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81104896 00000020:00010000:1.0:1789760793.701235:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.701238:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.701240:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.701243:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.701246:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.701252:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.701257:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.701275:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760793.701393:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 81100800<>81104896 00000100:00100000:0.0:1789760793.701514:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503924608:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760793.715936:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987107 gen 2 00000008:00010000:1.0:1789760793.715972:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:1.0:1789760793.715978:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.715987:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.716137:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000100:00100000:1.0:1789760793.716162:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503924608:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760793.716227:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19800, 19800] 00000080:00200000:3.0:1789760793.716239:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.716244:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.716255:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81104896 i_blocks 158432 00000020:00200000:3.0:1789760793.716258:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.716266:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.716291:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81104896, count: 4096 00000080:00200000:3.0:1789760793.716297:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81104896, bytes: 4096 00000080:00200000:3.0:1789760793.717292:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.717400:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760793.717402:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81104896, 81108992) 0 0 00000080:00200000:3.0:1789760793.717407:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.717424:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19801, 19801] 00020000:00200000:3.0:1789760793.717432:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81104896 chunk: [81104896, 81108992) 81108992, 4096 00020000:00200000:3.0:1789760793.717436:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.717462:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19801, 19801] 00000008:00010000:3.0:1789760793.717476:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.717481:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760793.717490:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81104896, 81108992) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.717501:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81104896=0x4d59000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.717554:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.717557:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81108992 00000020:00010000:3.0:1789760793.717563:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.717566:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.717570:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.717584:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.717587:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.717595:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.717600:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.717781:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1180 00000008:00100000:3.0:1789760793.717863:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 81104896<>81108992 00000100:00100000:3.0:1789760793.717913:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760793.717930:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503924736:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760793.741080:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987108 gen 2 00000008:00010000:3.0:1789760793.741114:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1180 00000020:00200000:3.0:1789760793.741120:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.741128:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.741146:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1180 00000100:00100000:3.0:1789760793.741166:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503924736:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760793.741248:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19801, 19801] 00000080:00200000:2.0:1789760793.741259:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.741265:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.741276:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81108992 i_blocks 158440 00000020:00200000:2.0:1789760793.741280:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.741287:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.741313:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81108992, count: 4096 00000080:00200000:2.0:1789760793.741319:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81108992, bytes: 4096 00000080:00200000:2.0:1789760793.741327:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.741335:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760793.741337:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81108992, 81113088) 0 0 00000080:00200000:2.0:1789760793.741340:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.741353:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19802, 19802] 00020000:00200000:2.0:1789760793.741359:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81108992 chunk: [81108992, 81113088) 81113088, 4096 00020000:00200000:2.0:1789760793.741363:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.741376:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19802, 19802] 00000008:00010000:2.0:1789760793.741390:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.741394:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760793.741402:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81108992, 81113088) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.741412:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81108992=0x4d5a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.741455:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.741459:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81113088 00000020:00010000:2.0:1789760793.741465:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.741468:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.741471:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.741474:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.741477:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.741484:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.741489:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.741502:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cc00 00000008:00100000:2.0:1789760793.741585:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25500 - object 0x280000401:2 offset 81108992<>81113088 00000100:00100000:3.0:1789760793.742022:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503924864:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760793.752493:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987109 gen 2 00000008:00010000:3.0:1789760793.752533:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760793.752539:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.752546:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.752569:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760793.752606:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503924864:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.752732:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19802, 19802] 00000080:00200000:1.0:1789760793.752744:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.752750:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.752761:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81113088 i_blocks 158448 00000020:00200000:1.0:1789760793.752765:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.752773:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.752797:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81113088, count: 4096 00000080:00200000:1.0:1789760793.752804:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81113088, bytes: 4096 00000080:00200000:1.0:1789760793.752812:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.752819:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.752821:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81113088, 81117184) 0 0 00000080:00200000:1.0:1789760793.752824:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.752836:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19803, 19803] 00020000:00200000:1.0:1789760793.752844:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81113088 chunk: [81113088, 81117184) 81117184, 4096 00020000:00200000:1.0:1789760793.752848:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.752861:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19803, 19803] 00000008:00010000:1.0:1789760793.752874:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.752878:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.752887:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81113088, 81117184) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.752898:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81113088=0x4d5b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.752934:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.752936:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81117184 00000020:00010000:1.0:1789760793.752943:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.752947:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.752950:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.752952:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.752955:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.752961:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.752966:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.753278:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:3.0:1789760793.753352:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 81113088<>81117184 00000100:00100000:2.0:1789760793.753835:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760793.753851:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503924992:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760793.769797:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987110 gen 2 00000008:00010000:3.0:1789760793.769971:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:3.0:1789760793.769979:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.769990:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.770137:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000100:00100000:3.0:1789760793.770163:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503924992:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.775741:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19803, 19803] 00000080:00200000:1.0:1789760793.775759:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.775764:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.775779:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81117184 i_blocks 158456 00000020:00200000:1.0:1789760793.775784:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.775794:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.775830:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81117184, count: 4096 00000080:00200000:1.0:1789760793.775838:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81117184, bytes: 4096 00000080:00200000:1.0:1789760793.775913:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.775922:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.775923:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81117184, 81121280) 0 0 00000080:00200000:1.0:1789760793.775927:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.775940:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19804, 19804] 00020000:00200000:1.0:1789760793.775946:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81117184 chunk: [81117184, 81121280) 81121280, 4096 00020000:00200000:1.0:1789760793.775950:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.775967:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19804, 19804] 00000008:00010000:1.0:1789760793.775980:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.775984:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.775993:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81117184, 81121280) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.776189:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81117184=0x4d5c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.776242:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.776245:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81121280 00000020:00010000:1.0:1789760793.776252:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.776255:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.776258:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.776261:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.776264:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.776271:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.776276:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.776297:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760793.776376:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 81117184<>81121280 00000100:00100000:0.0:1789760793.776454:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503925120:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.798277:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987111 gen 2 00000008:00010000:0.0:1789760793.798325:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58000 00000020:00200000:0.0:1789760793.798332:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.798340:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.798350:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58000 00000100:00100000:0.0:1789760793.798370:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503925120:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.798863:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19804, 19804] 00000080:00200000:1.0:1789760793.798883:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.798887:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.798899:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81121280 i_blocks 158464 00000020:00200000:1.0:1789760793.798902:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.798910:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.798936:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81121280, count: 4096 00000080:00200000:1.0:1789760793.798943:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81121280, bytes: 4096 00000080:00200000:1.0:1789760793.798951:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.798957:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.798959:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81121280, 81125376) 0 0 00000080:00200000:1.0:1789760793.798963:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.798975:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19805, 19805] 00020000:00200000:1.0:1789760793.798982:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81121280 chunk: [81121280, 81125376) 81125376, 4096 00020000:00200000:1.0:1789760793.798986:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.799000:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19805, 19805] 00000008:00010000:1.0:1789760793.799013:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.799018:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.799026:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81121280, 81125376) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.799064:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81121280=0x4d5d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.799106:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.799108:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81125376 00000020:00010000:1.0:1789760793.799114:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.799118:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.799121:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.799123:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.799126:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.799133:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.799138:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.799150:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760793.799225:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 81121280<>81125376 00000100:00100000:0.0:1789760793.799380:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503925248:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.821179:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987112 gen 2 00000008:00010000:0.0:1789760793.821216:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:0.0:1789760793.821222:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.821231:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.821251:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:0.0:1789760793.821275:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503925248:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.822478:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19805, 19805] 00000080:00200000:1.0:1789760793.822504:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.822510:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.822523:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81125376 i_blocks 158472 00000020:00200000:1.0:1789760793.822528:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.822537:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.822568:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81125376, count: 4096 00000080:00200000:1.0:1789760793.822583:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81125376, bytes: 4096 00000080:00200000:1.0:1789760793.822593:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.822602:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.822604:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81125376, 81129472) 0 0 00000080:00200000:1.0:1789760793.822608:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.822629:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19806, 19806] 00020000:00200000:1.0:1789760793.822636:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81125376 chunk: [81125376, 81129472) 81129472, 4096 00020000:00200000:1.0:1789760793.822639:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.822655:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19806, 19806] 00000008:00010000:1.0:1789760793.822670:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.822675:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.822683:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81125376, 81129472) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.822694:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81125376=0x4d5e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.822740:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.822742:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81129472 00000020:00010000:1.0:1789760793.822749:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.822752:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.822755:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.822757:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.822760:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.822765:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.822771:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.822785:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760793.823415:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 81125376<>81129472 00000100:00100000:0.0:1789760793.825752:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760793.825782:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503925376:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.849630:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987113 gen 2 00000008:00010000:0.0:1789760793.849666:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d586c0 00000020:00200000:0.0:1789760793.849673:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.849681:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.849702:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d586c0 00000100:00100000:0.0:1789760793.849725:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503925376:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.849969:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19806, 19806] 00000080:00200000:1.0:1789760793.849982:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.849987:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.850000:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81129472 i_blocks 158480 00000020:00200000:1.0:1789760793.850003:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.850012:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.850072:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81129472, count: 4096 00000080:00200000:1.0:1789760793.850080:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81129472, bytes: 4096 00000080:00200000:1.0:1789760793.850087:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.850094:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.850096:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81129472, 81133568) 0 0 00000080:00200000:1.0:1789760793.850100:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.850112:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19807, 19807] 00020000:00200000:1.0:1789760793.850118:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81129472 chunk: [81129472, 81133568) 81133568, 4096 00020000:00200000:1.0:1789760793.850121:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.850136:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19807, 19807] 00000008:00010000:1.0:1789760793.850149:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.850153:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.850162:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81129472, 81133568) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.850172:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81129472=0x4d5f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.850215:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.850218:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81133568 00000020:00010000:1.0:1789760793.850224:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.850227:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.850230:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.850233:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.850236:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.850241:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.850247:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.850261:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760793.850351:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81129472<>81133568 00000100:00100000:0.0:1789760793.850429:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760793.850444:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503925504:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.866146:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987114 gen 2 00000008:00010000:0.0:1789760793.866184:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58a80 00000020:00200000:0.0:1789760793.866190:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.866199:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.866219:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58a80 00000100:00100000:0.0:1789760793.866242:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503925504:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.866520:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19807, 19807] 00000080:00200000:1.0:1789760793.866532:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.866536:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.866548:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81133568 i_blocks 158488 00000020:00200000:1.0:1789760793.866552:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.866559:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.866594:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81133568, count: 4096 00000080:00200000:1.0:1789760793.866601:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81133568, bytes: 4096 00000080:00200000:1.0:1789760793.866608:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.867725:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.867729:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81133568, 81137664) 0 0 00000080:00200000:1.0:1789760793.867735:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.867749:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19808, 19808] 00020000:00200000:1.0:1789760793.867758:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81133568 chunk: [81133568, 81137664) 81137664, 4096 00020000:00200000:1.0:1789760793.867762:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.867778:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19808, 19808] 00000008:00010000:1.0:1789760793.867795:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.867799:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.867809:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81133568, 81137664) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.867821:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81133568=0x4d60000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.867872:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.867875:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81137664 00000020:00010000:1.0:1789760793.867882:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.867885:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.867888:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.867891:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.867893:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.867900:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.867906:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.867927:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760793.868006:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81133568<>81137664 00000100:00100000:0.0:1789760793.868407:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503925632:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.883100:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987115 gen 2 00000008:00010000:0.0:1789760793.883135:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58540 00000020:00200000:0.0:1789760793.883142:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.883150:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.883160:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58540 00000100:00100000:0.0:1789760793.883182:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503925632:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.883350:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19808, 19808] 00000080:00200000:1.0:1789760793.883363:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.883368:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.883380:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81137664 i_blocks 158496 00000020:00200000:1.0:1789760793.883384:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.883391:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.883417:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81137664, count: 4096 00000080:00200000:1.0:1789760793.883424:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81137664, bytes: 4096 00000080:00200000:1.0:1789760793.883431:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.883438:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.883440:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81137664, 81141760) 0 0 00000080:00200000:1.0:1789760793.883444:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.883457:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19809, 19809] 00020000:00200000:1.0:1789760793.883464:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81137664 chunk: [81137664, 81141760) 81141760, 4096 00020000:00200000:1.0:1789760793.883468:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.883483:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19809, 19809] 00000008:00010000:1.0:1789760793.883693:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.883700:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.883710:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81137664, 81141760) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.883722:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81137664=0x4d61000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.883772:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.883775:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81141760 00000020:00010000:1.0:1789760793.883783:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.883786:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.883789:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.883793:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.883796:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.883803:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.883809:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.883824:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760793.883907:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 81137664<>81141760 00000100:00100000:0.0:1789760793.884009:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503925760:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.893725:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987116 gen 2 00000008:00010000:0.0:1789760793.893761:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d583c0 00000020:00200000:0.0:1789760793.893766:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.893774:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.893792:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d583c0 00000100:00100000:0.0:1789760793.893812:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503925760:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.893901:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19809, 19809] 00000080:00200000:1.0:1789760793.893913:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.893918:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.893929:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81141760 i_blocks 158504 00000020:00200000:1.0:1789760793.893933:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.893941:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.893966:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81141760, count: 4096 00000080:00200000:1.0:1789760793.893971:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81141760, bytes: 4096 00000080:00200000:1.0:1789760793.893978:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.893984:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.893986:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81141760, 81145856) 0 0 00000080:00200000:1.0:1789760793.893990:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.894002:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19810, 19810] 00020000:00200000:1.0:1789760793.894009:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81141760 chunk: [81141760, 81145856) 81145856, 4096 00020000:00200000:1.0:1789760793.894012:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.894026:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19810, 19810] 00000008:00010000:1.0:1789760793.894061:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.894066:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.894072:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81141760, 81145856) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.894082:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81141760=0x4d62000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.894119:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.894121:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81145856 00000020:00010000:1.0:1789760793.894127:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.894130:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.894133:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.894135:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.894138:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.894144:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.894148:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.894160:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760793.894234:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 81141760<>81145856 00000100:00100000:0.0:1789760793.894431:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503925888:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.913298:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987117 gen 2 00000008:00010000:0.0:1789760793.913334:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:0.0:1789760793.913340:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.913351:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.913370:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:0.0:1789760793.913394:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503925888:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.913492:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19810, 19810] 00000080:00200000:1.0:1789760793.913505:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.913510:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.913523:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81145856 i_blocks 158512 00000020:00200000:1.0:1789760793.913527:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.913535:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.913559:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81145856, count: 4096 00000080:00200000:1.0:1789760793.913566:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81145856, bytes: 4096 00000080:00200000:1.0:1789760793.913584:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.913592:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.913594:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81145856, 81149952) 0 0 00000080:00200000:1.0:1789760793.913598:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.913620:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19811, 19811] 00020000:00200000:1.0:1789760793.913633:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81145856 chunk: [81145856, 81149952) 81149952, 4096 00020000:00200000:1.0:1789760793.913637:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.913652:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19811, 19811] 00000008:00010000:1.0:1789760793.913665:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.913670:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.913678:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81145856, 81149952) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.913689:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81145856=0x4d63000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.913735:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.913738:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81149952 00000020:00010000:1.0:1789760793.913744:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.913747:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.913750:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.913754:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.913756:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.913762:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.913768:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.913781:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760793.913857:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 81145856<>81149952 00000100:00100000:0.0:1789760793.914059:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503926016:192.168.203.158@tcp:4:dd.0 00010000:00010000:3.0:1789760793.918738:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760793.928858:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987118 gen 2 00000008:00010000:0.0:1789760793.928895:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:0.0:1789760793.928902:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.928911:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.928934:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:0.0:1789760793.928959:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503926016:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.929846:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19811, 19811] 00000080:00200000:1.0:1789760793.929862:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.929868:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.929881:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81149952 i_blocks 158520 00000020:00200000:1.0:1789760793.929885:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.929894:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.929920:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81149952, count: 4096 00000080:00200000:1.0:1789760793.929927:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81149952, bytes: 4096 00000080:00200000:1.0:1789760793.929937:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.929945:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.929947:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81149952, 81154048) 0 0 00000080:00200000:1.0:1789760793.929951:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.929963:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19812, 19812] 00020000:00200000:1.0:1789760793.929970:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81149952 chunk: [81149952, 81154048) 81154048, 4096 00020000:00200000:1.0:1789760793.929974:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.929988:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19812, 19812] 00000008:00010000:1.0:1789760793.930002:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.930007:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.930017:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81149952, 81154048) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.930028:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81149952=0x4d64000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.930100:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.930104:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81154048 00000020:00010000:1.0:1789760793.930110:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.930113:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.930128:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.930132:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.930134:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.930141:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.930147:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.930162:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760793.930239:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 81149952<>81154048 00000100:00100000:0.0:1789760793.930973:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503926144:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.953990:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987119 gen 2 00000008:00010000:0.0:1789760793.954026:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d583c0 00000020:00200000:0.0:1789760793.954032:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.954074:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.954083:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d583c0 00000100:00100000:0.0:1789760793.954104:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503926144:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.954801:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19812, 19812] 00000080:00200000:1.0:1789760793.954814:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.954818:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.954832:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81154048 i_blocks 158528 00000020:00200000:1.0:1789760793.954835:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.954842:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.954866:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81154048, count: 4096 00000080:00200000:1.0:1789760793.954872:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81154048, bytes: 4096 00000080:00200000:1.0:1789760793.954881:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.954888:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.954890:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81154048, 81158144) 0 0 00000080:00200000:1.0:1789760793.954893:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.954905:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19813, 19813] 00020000:00200000:1.0:1789760793.954912:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81154048 chunk: [81154048, 81158144) 81158144, 4096 00020000:00200000:1.0:1789760793.954915:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.954929:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19813, 19813] 00000008:00010000:1.0:1789760793.954942:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.954946:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.954954:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81154048, 81158144) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.954965:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81154048=0x4d65000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.955009:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.955012:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81158144 00000020:00010000:1.0:1789760793.955019:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.955022:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.955025:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.955028:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.955031:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.955072:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.955077:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.955091:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760793.955174:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 81154048<>81158144 00000100:00100000:0.0:1789760793.955260:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503926272:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.973819:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987120 gen 2 00000008:00010000:0.0:1789760793.973856:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58540 00000020:00200000:0.0:1789760793.973863:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.973872:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.973891:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58540 00000100:00100000:0.0:1789760793.973915:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503926272:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.974123:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19813, 19813] 00000080:00200000:1.0:1789760793.974136:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.974141:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.974153:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81158144 i_blocks 158536 00000020:00200000:1.0:1789760793.974157:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.974164:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.974192:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81158144, count: 4096 00000080:00200000:1.0:1789760793.974199:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81158144, bytes: 4096 00000080:00200000:1.0:1789760793.974206:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.974213:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.974215:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81158144, 81162240) 0 0 00000080:00200000:1.0:1789760793.974219:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.974231:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19814, 19814] 00020000:00200000:1.0:1789760793.974237:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81158144 chunk: [81158144, 81162240) 81162240, 4096 00020000:00200000:1.0:1789760793.974241:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.974256:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19814, 19814] 00000008:00010000:1.0:1789760793.974269:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.974274:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.974283:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81158144, 81162240) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.974371:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81158144=0x4d66000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.974421:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.974424:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81162240 00000020:00010000:1.0:1789760793.974432:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.974434:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.974437:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.974440:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.974442:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.974449:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.974455:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.974470:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760793.974706:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 81158144<>81162240 00000100:00100000:0.0:1789760793.974767:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503926400:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760793.992810:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987121 gen 2 00000008:00010000:0.0:1789760793.992845:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58a80 00000020:00200000:0.0:1789760793.992852:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.992860:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.992878:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58a80 00000100:00100000:0.0:1789760793.992903:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503926400:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760793.993010:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19814, 19814] 00000080:00200000:1.0:1789760793.993023:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.993028:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.993064:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81162240 i_blocks 158544 00000020:00200000:1.0:1789760793.993068:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.993076:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.993102:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81162240, count: 4096 00000080:00200000:1.0:1789760793.993108:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81162240, bytes: 4096 00000080:00200000:1.0:1789760793.993116:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.993122:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760793.993124:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81162240, 81166336) 0 0 00000080:00200000:1.0:1789760793.993128:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.993138:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19815, 19815] 00020000:00200000:1.0:1789760793.993145:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81162240 chunk: [81162240, 81166336) 81166336, 4096 00020000:00200000:1.0:1789760793.993149:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.993162:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19815, 19815] 00000008:00010000:1.0:1789760793.993175:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.993179:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760793.993187:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81162240, 81166336) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.993198:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81162240=0x4d67000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.993302:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.993305:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81166336 00000020:00010000:1.0:1789760793.993312:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.993315:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.993318:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.993321:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.993324:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.993331:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.993337:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.993352:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760793.993426:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81162240<>81166336 00000100:00100000:0.0:1789760793.993863:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503926528:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.008220:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987122 gen 2 00000008:00010000:0.0:1789760794.008254:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d586c0 00000020:00200000:0.0:1789760794.008261:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.008269:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.008287:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d586c0 00000100:00100000:0.0:1789760794.008312:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503926528:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.009103:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19815, 19815] 00000080:00200000:1.0:1789760794.009117:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.009123:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.009136:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81166336 i_blocks 158552 00000020:00200000:1.0:1789760794.009140:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.009148:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.009173:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81166336, count: 4096 00000080:00200000:1.0:1789760794.009179:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81166336, bytes: 4096 00000080:00200000:1.0:1789760794.009187:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.009195:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.009196:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81166336, 81170432) 0 0 00000080:00200000:1.0:1789760794.009200:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.009212:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19816, 19816] 00020000:00200000:1.0:1789760794.009218:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81166336 chunk: [81166336, 81170432) 81170432, 4096 00020000:00200000:1.0:1789760794.009222:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.009235:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19816, 19816] 00000008:00010000:1.0:1789760794.009248:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.009253:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.009261:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81166336, 81170432) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.009273:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81166336=0x4d68000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.009316:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.009319:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81170432 00000020:00010000:1.0:1789760794.010434:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.010443:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.010448:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.010451:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.010454:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.010464:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.010473:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.010846:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760794.011088:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81166336<>81170432 00000100:00100000:1.0:1789760794.011133:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760794.011153:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503926656:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.028024:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987123 gen 2 00000008:00010000:1.0:1789760794.028093:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:1.0:1789760794.028100:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.028109:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.028130:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:1.0:1789760794.028153:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503926656:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.029753:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19816, 19816] 00000080:00200000:0.0:1789760794.029770:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.029775:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.029789:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81170432 i_blocks 158560 00000020:00200000:0.0:1789760794.029794:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.029803:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.029829:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81170432, count: 4096 00000080:00200000:0.0:1789760794.029834:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81170432, bytes: 4096 00000080:00200000:0.0:1789760794.029844:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.029851:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.029853:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81170432, 81174528) 0 0 00000080:00200000:0.0:1789760794.029857:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.029881:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19817, 19817] 00020000:00200000:0.0:1789760794.029888:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81170432 chunk: [81170432, 81174528) 81174528, 4096 00020000:00200000:0.0:1789760794.029892:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.029908:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19817, 19817] 00000008:00010000:0.0:1789760794.029923:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.029928:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.030087:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81170432, 81174528) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.030102:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81170432=0x4d69000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.030155:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.030157:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81174528 00000020:00010000:0.0:1789760794.030164:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.030168:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.030170:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.030174:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.030176:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.030184:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.030190:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.030206:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760794.030748:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a300 - object 0x280000401:2 offset 81170432<>81174528 00000100:00100000:0.0:1789760794.030787:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760794.030807:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503926784:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.049478:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987124 gen 2 00000008:00010000:0.0:1789760794.049523:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:0.0:1789760794.049531:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.049541:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.049561:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:0.0:1789760794.049596:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503926784:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.049699:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19817, 19817] 00000080:00200000:1.0:1789760794.049711:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.049717:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.049728:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81174528 i_blocks 158568 00000020:00200000:1.0:1789760794.049732:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.049740:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.049766:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81174528, count: 4096 00000080:00200000:1.0:1789760794.049772:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81174528, bytes: 4096 00000080:00200000:1.0:1789760794.049780:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.049787:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.049789:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81174528, 81178624) 0 0 00000080:00200000:1.0:1789760794.049793:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.049806:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19818, 19818] 00020000:00200000:1.0:1789760794.049814:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81174528 chunk: [81174528, 81178624) 81178624, 4096 00020000:00200000:1.0:1789760794.049818:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.049832:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19818, 19818] 00000008:00010000:1.0:1789760794.049847:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.049853:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.049872:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81174528, 81178624) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.049884:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81174528=0x4d6a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.049930:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.049944:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81178624 00000020:00010000:1.0:1789760794.049951:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.049954:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.049957:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.049961:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.049964:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.049970:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.049977:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.049991:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760794.050106:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81174528<>81178624 00000100:00100000:1.0:1789760794.050131:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760794.050143:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503926912:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.082753:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987125 gen 2 00000008:00010000:1.0:1789760794.082786:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:1.0:1789760794.082792:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.082800:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.082823:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:1.0:1789760794.082847:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503926912:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760794.082945:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19818, 19818] 00000080:00200000:3.0:1789760794.082958:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.082963:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.082975:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81178624 i_blocks 158576 00000020:00200000:3.0:1789760794.082978:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.082987:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.083014:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81178624, count: 4096 00000080:00200000:3.0:1789760794.083019:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81178624, bytes: 4096 00000080:00200000:3.0:1789760794.083028:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.083073:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760794.083076:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81178624, 81182720) 0 0 00000080:00200000:3.0:1789760794.083080:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.083099:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19819, 19819] 00020000:00200000:3.0:1789760794.083106:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81178624 chunk: [81178624, 81182720) 81182720, 4096 00020000:00200000:3.0:1789760794.083109:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.083124:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19819, 19819] 00000008:00010000:3.0:1789760794.083137:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.083141:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760794.083150:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81178624, 81182720) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.083160:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81178624=0x4d6b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.083202:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.083204:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81182720 00000020:00010000:3.0:1789760794.083209:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.083213:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.083215:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.083218:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.083221:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.083227:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.083232:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.083244:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1180 00000008:00100000:3.0:1789760794.083311:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 81178624<>81182720 00000100:00100000:2.0:1789760794.083435:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760794.083450:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503927040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.099096:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987126 gen 2 00000008:00010000:3.0:1789760794.099143:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1180 00000020:00200000:3.0:1789760794.099149:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.099158:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.099177:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1180 00000100:00100000:3.0:1789760794.099200:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503927040:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.099464:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19819, 19819] 00000080:00200000:1.0:1789760794.099477:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.099482:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.099494:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81182720 i_blocks 158584 00000020:00200000:1.0:1789760794.099497:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.099505:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.099532:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81182720, count: 4096 00000080:00200000:1.0:1789760794.099538:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81182720, bytes: 4096 00000080:00200000:1.0:1789760794.099546:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.099553:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.099555:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81182720, 81186816) 0 0 00000080:00200000:1.0:1789760794.099559:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.099580:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19820, 19820] 00020000:00200000:1.0:1789760794.099587:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81182720 chunk: [81182720, 81186816) 81186816, 4096 00020000:00200000:1.0:1789760794.099591:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.099606:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19820, 19820] 00000008:00010000:1.0:1789760794.099682:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.099688:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.099697:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81182720, 81186816) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.099707:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81182720=0x4d6c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.099751:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.099754:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81186816 00000020:00010000:1.0:1789760794.099761:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.099764:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.099766:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.099769:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.099771:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.099777:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.099783:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.099797:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760794.099874:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81182720<>81186816 00000100:00100000:0.0:1789760794.099941:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503927168:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.114682:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987127 gen 2 00000008:00010000:0.0:1789760794.114721:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:0.0:1789760794.114728:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.114737:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.114758:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:0.0:1789760794.114785:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503927168:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760794.114964:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19820, 19820] 00000080:00200000:3.0:1789760794.114976:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.114981:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.114992:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81186816 i_blocks 158592 00000020:00200000:3.0:1789760794.114996:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.115003:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.115026:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81186816, count: 4096 00000080:00200000:3.0:1789760794.115059:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81186816, bytes: 4096 00000080:00200000:3.0:1789760794.115071:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.115078:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760794.115080:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81186816, 81190912) 0 0 00000080:00200000:3.0:1789760794.115084:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.115097:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19821, 19821] 00020000:00200000:3.0:1789760794.115103:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81186816 chunk: [81186816, 81190912) 81190912, 4096 00020000:00200000:3.0:1789760794.115107:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.115122:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19821, 19821] 00000008:00010000:3.0:1789760794.115136:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.115140:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760794.115147:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81186816, 81190912) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.115159:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81186816=0x4d6d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.115201:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.115203:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81190912 00000020:00010000:3.0:1789760794.115210:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.115213:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.115215:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.115219:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.115221:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.115227:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.115234:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.115248:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1180 00000008:00100000:3.0:1789760794.115328:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 81186816<>81190912 00000100:00100000:2.0:1789760794.115736:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503927296:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.136231:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987128 gen 2 00000008:00010000:3.0:1789760794.136271:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1180 00000020:00200000:3.0:1789760794.136278:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.136288:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.136309:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1180 00000100:00100000:3.0:1789760794.136333:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503927296:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.136442:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19821, 19821] 00000080:00200000:1.0:1789760794.136455:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.136460:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.136471:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81190912 i_blocks 158600 00000020:00200000:1.0:1789760794.136474:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.136481:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.136508:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81190912, count: 4096 00000080:00200000:1.0:1789760794.136514:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81190912, bytes: 4096 00000080:00200000:1.0:1789760794.136522:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.136530:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.136532:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81190912, 81195008) 0 0 00000080:00200000:1.0:1789760794.136535:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.136547:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19822, 19822] 00020000:00200000:1.0:1789760794.136554:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81190912 chunk: [81190912, 81195008) 81195008, 4096 00020000:00200000:1.0:1789760794.136557:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.136581:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19822, 19822] 00000008:00010000:1.0:1789760794.136596:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.136600:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.136619:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81190912, 81195008) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.136631:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81190912=0x4d6e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.136680:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.136682:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81195008 00000020:00010000:1.0:1789760794.136689:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.136692:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.136695:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.136698:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.136701:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.136707:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.136713:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.136727:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760794.136801:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 81190912<>81195008 00000100:00100000:0.0:1789760794.136964:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503927424:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.153352:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987129 gen 2 00000008:00010000:0.0:1789760794.153389:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58000 00000020:00200000:0.0:1789760794.153396:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.153405:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.153426:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58000 00000100:00100000:0.0:1789760794.153452:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503927424:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.153771:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19822, 19822] 00000080:00200000:1.0:1789760794.153785:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.153790:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.153812:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81195008 i_blocks 158608 00000020:00200000:1.0:1789760794.153815:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.153824:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.153850:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81195008, count: 4096 00000080:00200000:1.0:1789760794.153856:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81195008, bytes: 4096 00000080:00200000:1.0:1789760794.153865:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.153873:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.153875:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81195008, 81199104) 0 0 00000080:00200000:1.0:1789760794.153878:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.153892:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19823, 19823] 00020000:00200000:1.0:1789760794.153900:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81195008 chunk: [81195008, 81199104) 81199104, 4096 00020000:00200000:1.0:1789760794.153904:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.153920:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19823, 19823] 00000008:00010000:1.0:1789760794.153934:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.153939:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.153948:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81195008, 81199104) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.153960:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81195008=0x4d6f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.154006:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.154009:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81199104 00000020:00010000:1.0:1789760794.154015:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.154018:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.154021:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.154024:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.154026:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.154073:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.154080:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.154093:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760794.154174:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 81195008<>81199104 00000100:00100000:0.0:1789760794.154259:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760794.154275:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503927552:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.167212:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987130 gen 2 00000008:00010000:0.0:1789760794.167246:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:0.0:1789760794.167252:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.167261:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.167282:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000100:00100000:0.0:1789760794.167305:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503927552:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.167456:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19823, 19823] 00000080:00200000:1.0:1789760794.167469:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.167474:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.167487:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81199104 i_blocks 158616 00000020:00200000:1.0:1789760794.167491:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.167500:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.167524:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81199104, count: 4096 00000080:00200000:1.0:1789760794.167530:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81199104, bytes: 4096 00000080:00200000:1.0:1789760794.167538:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.167545:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.167548:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81199104, 81203200) 0 0 00000080:00200000:1.0:1789760794.167551:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.167564:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19824, 19824] 00020000:00200000:1.0:1789760794.167581:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81199104 chunk: [81199104, 81203200) 81203200, 4096 00020000:00200000:1.0:1789760794.167585:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.167600:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19824, 19824] 00000008:00010000:1.0:1789760794.167625:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.167631:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.167640:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81199104, 81203200) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.167651:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81199104=0x4d70000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.167699:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.167701:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81203200 00000020:00010000:1.0:1789760794.167708:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.167712:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.167714:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.167717:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.167720:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.167727:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.167732:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.167746:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58780 00000008:00100000:1.0:1789760794.167826:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 81199104<>81203200 00000100:00100000:0.0:1789760794.168082:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503927680:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.179370:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987131 gen 2 00000008:00010000:0.0:1789760794.179405:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58780 00000020:00200000:0.0:1789760794.179412:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.179420:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.179442:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58780 00000100:00100000:0.0:1789760794.179465:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503927680:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.179471:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19824, 19824] 00000080:00200000:1.0:1789760794.179485:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.179489:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.179501:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81203200 i_blocks 158624 00000020:00200000:1.0:1789760794.179505:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.179512:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.179536:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81203200, count: 4096 00000080:00200000:1.0:1789760794.179543:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81203200, bytes: 4096 00000080:00200000:1.0:1789760794.179550:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.179558:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.179560:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81203200, 81207296) 0 0 00000080:00200000:1.0:1789760794.179563:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.179585:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19825, 19825] 00020000:00200000:1.0:1789760794.179592:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81203200 chunk: [81203200, 81207296) 81207296, 4096 00020000:00200000:1.0:1789760794.179596:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.179620:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19825, 19825] 00000008:00010000:1.0:1789760794.179635:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.179641:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.179649:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81203200, 81207296) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.179659:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81203200=0x4d71000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.179703:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.179706:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81207296 00000020:00010000:1.0:1789760794.179712:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.179715:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.179718:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.179721:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.179723:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.179729:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.179735:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.179748:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760794.179832:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 81203200<>81207296 00000100:00100000:0.0:1789760794.179894:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503927808:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.190834:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987132 gen 2 00000008:00010000:0.0:1789760794.190869:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:0.0:1789760794.190875:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.190885:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.190906:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:0.0:1789760794.190930:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503927808:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.191030:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19825, 19825] 00000080:00200000:1.0:1789760794.191060:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.191065:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.191077:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81207296 i_blocks 158632 00000020:00200000:1.0:1789760794.191081:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.191088:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.191112:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81207296, count: 4096 00000080:00200000:1.0:1789760794.191118:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81207296, bytes: 4096 00000080:00200000:1.0:1789760794.191125:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.191131:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.191133:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81207296, 81211392) 0 0 00000080:00200000:1.0:1789760794.191137:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.191149:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19826, 19826] 00020000:00200000:1.0:1789760794.191160:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81207296 chunk: [81207296, 81211392) 81211392, 4096 00020000:00200000:1.0:1789760794.191164:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.191177:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19826, 19826] 00000008:00010000:1.0:1789760794.191191:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.191196:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.191204:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81207296, 81211392) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.191214:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81207296=0x4d72000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.191258:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.191260:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81211392 00000020:00010000:1.0:1789760794.191267:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.191270:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.191273:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.191276:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.191278:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.191284:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.191291:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.191304:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760794.191394:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 81207296<>81211392 00000100:00100000:0.0:1789760794.191449:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503927936:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.199089:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987133 gen 2 00000008:00010000:0.0:1789760794.199127:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760794.199133:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.199141:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.199155:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760794.199178:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503927936:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.199344:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19826, 19826] 00000080:00200000:1.0:1789760794.199356:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.199362:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.199374:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81211392 i_blocks 158640 00000020:00200000:1.0:1789760794.199377:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.199384:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.199410:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81211392, count: 4096 00000080:00200000:1.0:1789760794.199415:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81211392, bytes: 4096 00000080:00200000:1.0:1789760794.199422:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.199428:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.199431:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81211392, 81215488) 0 0 00000080:00200000:1.0:1789760794.199434:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.199447:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19827, 19827] 00020000:00200000:1.0:1789760794.199453:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81211392 chunk: [81211392, 81215488) 81215488, 4096 00020000:00200000:1.0:1789760794.199457:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.199471:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19827, 19827] 00000008:00010000:1.0:1789760794.199484:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.199489:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.199495:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81211392, 81215488) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.199506:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81211392=0x4d73000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.199547:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.199550:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81215488 00000020:00010000:1.0:1789760794.199557:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.199560:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.199562:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.199565:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.199568:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.199583:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.199589:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.199601:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760794.199780:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 81211392<>81215488 00000100:00100000:0.0:1789760794.199892:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503928064:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.209239:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987134 gen 2 00000008:00010000:0.0:1789760794.209273:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760794.209279:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.209288:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.209424:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:0.0:1789760794.209448:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503928064:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.209742:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19827, 19827] 00000080:00200000:1.0:1789760794.209753:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.209759:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.209771:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81215488 i_blocks 158648 00000020:00200000:1.0:1789760794.209775:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.209783:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.209808:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81215488, count: 4096 00000080:00200000:1.0:1789760794.209814:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81215488, bytes: 4096 00000080:00200000:1.0:1789760794.209821:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.209828:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.209829:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81215488, 81219584) 0 0 00000080:00200000:1.0:1789760794.209833:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.209845:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19828, 19828] 00020000:00200000:1.0:1789760794.209852:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81215488 chunk: [81215488, 81219584) 81219584, 4096 00020000:00200000:1.0:1789760794.209855:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.209870:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19828, 19828] 00000008:00010000:1.0:1789760794.209882:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.209887:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.209897:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81215488, 81219584) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.209907:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81215488=0x4d74000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.209951:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.209953:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81219584 00000020:00010000:1.0:1789760794.209960:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.209962:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.209966:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.209969:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.209971:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.209977:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.209982:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.209996:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760794.210245:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81215488<>81219584 00000100:00100000:0.0:1789760794.210362:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503928192:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.221991:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987135 gen 2 00000008:00010000:0.0:1789760794.222025:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760794.222032:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.222190:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.222204:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000100:00100000:0.0:1789760794.222229:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503928192:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760794.222891:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19828, 19828] 00000080:00200000:3.0:1789760794.222905:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.222910:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.222923:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81219584 i_blocks 158656 00000020:00200000:3.0:1789760794.222926:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.222934:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.222958:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81219584, count: 4096 00000080:00200000:3.0:1789760794.222964:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81219584, bytes: 4096 00000080:00200000:3.0:1789760794.222972:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.222980:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760794.222982:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81219584, 81223680) 0 0 00000080:00200000:3.0:1789760794.222985:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.222998:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19829, 19829] 00020000:00200000:3.0:1789760794.223005:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81219584 chunk: [81219584, 81223680) 81223680, 4096 00020000:00200000:3.0:1789760794.223025:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.223058:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19829, 19829] 00000008:00010000:3.0:1789760794.223072:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.223076:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760794.223084:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81219584, 81223680) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.223096:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81219584=0x4d75000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.223138:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.223141:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81223680 00000020:00010000:3.0:1789760794.223147:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.223151:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.223153:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.223157:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.223159:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.223165:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.223171:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.223185:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1180 00000008:00100000:3.0:1789760794.223267:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 81219584<>81223680 00000100:00100000:3.0:1789760794.223309:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760794.223322:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503928320:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.231253:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987136 gen 2 00000008:00010000:3.0:1789760794.231288:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1180 00000020:00200000:3.0:1789760794.231294:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.231302:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.231358:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1180 00000100:00100000:3.0:1789760794.231382:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503928320:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.231488:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19829, 19829] 00000080:00200000:2.0:1789760794.231499:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.231505:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.231516:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81223680 i_blocks 158664 00000020:00200000:2.0:1789760794.231519:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.231527:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.231551:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81223680, count: 4096 00000080:00200000:2.0:1789760794.231557:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81223680, bytes: 4096 00000080:00200000:2.0:1789760794.231564:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.231583:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.231585:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81223680, 81227776) 0 0 00000080:00200000:2.0:1789760794.231589:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.231602:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19830, 19830] 00020000:00200000:2.0:1789760794.231609:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81223680 chunk: [81223680, 81227776) 81227776, 4096 00020000:00200000:2.0:1789760794.231612:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.231627:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19830, 19830] 00000008:00010000:2.0:1789760794.231641:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.231645:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.231652:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81223680, 81227776) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.231662:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81223680=0x4d76000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.231703:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.231706:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81227776 00000020:00010000:2.0:1789760794.231712:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.231715:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.231718:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.231726:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.231729:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.231735:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.231740:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.231753:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c300 00000008:00100000:2.0:1789760794.231818:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24000 - object 0x280000401:2 offset 81223680<>81227776 00000100:00100000:2.0:1789760794.231845:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760794.231859:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503928448:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.243885:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987137 gen 2 00000008:00010000:3.0:1789760794.243919:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760794.244173:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.244182:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.244201:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760794.244225:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503928448:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.244319:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19830, 19830] 00000080:00200000:2.0:1789760794.244332:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.244337:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.244349:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81227776 i_blocks 158672 00000020:00200000:2.0:1789760794.244352:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.244360:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.244385:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81227776, count: 4096 00000080:00200000:2.0:1789760794.244391:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81227776, bytes: 4096 00000080:00200000:2.0:1789760794.244398:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.244405:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.244407:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81227776, 81231872) 0 0 00000080:00200000:2.0:1789760794.244410:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.244423:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19831, 19831] 00020000:00200000:2.0:1789760794.244429:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81227776 chunk: [81227776, 81231872) 81231872, 4096 00020000:00200000:2.0:1789760794.244433:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.244447:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19831, 19831] 00000008:00010000:2.0:1789760794.244460:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.244465:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.244475:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81227776, 81231872) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.244486:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81227776=0x4d77000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.244540:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.244542:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81231872 00000020:00010000:2.0:1789760794.244549:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.244553:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.244556:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.244567:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.244803:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.244811:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.244818:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.244832:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000008:00100000:2.0:1789760794.244904:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24000 - object 0x280000401:2 offset 81227776<>81231872 00000100:00100000:3.0:1789760794.245081:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503928576:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.258401:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987138 gen 2 00000008:00010000:3.0:1789760794.258440:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760794.258447:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.258457:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.260129:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760794.260167:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503928576:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.261916:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19831, 19831] 00000080:00200000:2.0:1789760794.261935:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.261941:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.261954:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81231872 i_blocks 158680 00000020:00200000:2.0:1789760794.261959:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.261969:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.261995:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81231872, count: 4096 00000080:00200000:2.0:1789760794.262002:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81231872, bytes: 4096 00000080:00200000:2.0:1789760794.262013:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.262020:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.262022:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81231872, 81235968) 0 0 00000080:00200000:2.0:1789760794.262026:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.262073:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19832, 19832] 00020000:00200000:2.0:1789760794.262080:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81231872 chunk: [81231872, 81235968) 81235968, 4096 00020000:00200000:2.0:1789760794.262085:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.262102:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19832, 19832] 00000008:00010000:2.0:1789760794.262116:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.262121:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.262130:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81231872, 81235968) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.262141:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81231872=0x4d78000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.262190:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.262193:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81235968 00000020:00010000:2.0:1789760794.262200:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.262203:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.262206:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.262209:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.262211:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.262219:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.262224:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.262240:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000008:00100000:2.0:1789760794.262320:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25500 - object 0x280000401:2 offset 81231872<>81235968 00000100:00100000:3.0:1789760794.262738:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503928704:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.278227:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987139 gen 2 00000008:00010000:3.0:1789760794.278263:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760794.278270:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.278278:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.278299:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760794.278323:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503928704:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.278405:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19832, 19832] 00000080:00200000:2.0:1789760794.278417:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.278422:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.278434:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81235968 i_blocks 158688 00000020:00200000:2.0:1789760794.278437:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.278444:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.278467:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81235968, count: 4096 00000080:00200000:2.0:1789760794.278473:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81235968, bytes: 4096 00000080:00200000:2.0:1789760794.278480:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.278488:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.278489:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81235968, 81240064) 0 0 00000080:00200000:2.0:1789760794.278493:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.278505:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19833, 19833] 00020000:00200000:2.0:1789760794.278512:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81235968 chunk: [81235968, 81240064) 81240064, 4096 00020000:00200000:2.0:1789760794.278516:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.278528:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19833, 19833] 00000008:00010000:2.0:1789760794.278540:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.278545:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.278554:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81235968, 81240064) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.278592:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81235968=0x4d79000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.278635:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.278638:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81240064 00000020:00010000:2.0:1789760794.278645:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.278648:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.278651:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.278654:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.278657:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.278664:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.278669:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.278682:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c480 00000008:00100000:2.0:1789760794.278753:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26680 - object 0x280000401:2 offset 81235968<>81240064 00000100:00100000:3.0:1789760794.279060:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760794.279074:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503928832:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.287975:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987140 gen 2 00000008:00010000:3.0:1789760794.288009:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760794.288016:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.288025:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.288166:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760794.288192:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503928832:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.289313:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19833, 19833] 00000080:00200000:2.0:1789760794.289327:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.289332:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.289343:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81240064 i_blocks 158696 00000020:00200000:2.0:1789760794.289346:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.289355:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.289377:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81240064, count: 4096 00000080:00200000:2.0:1789760794.289382:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81240064, bytes: 4096 00000080:00200000:2.0:1789760794.289390:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.289397:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.289399:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81240064, 81244160) 0 0 00000080:00200000:2.0:1789760794.289403:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.289413:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19834, 19834] 00020000:00200000:2.0:1789760794.289419:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81240064 chunk: [81240064, 81244160) 81244160, 4096 00020000:00200000:2.0:1789760794.289423:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.289436:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19834, 19834] 00000008:00010000:2.0:1789760794.289448:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.289453:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.289461:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81240064, 81244160) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.289471:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81240064=0x4d7a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.289590:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.289593:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81244160 00000020:00010000:2.0:1789760794.289600:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.289603:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.289605:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.289609:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.289611:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.289618:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.289624:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.289642:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c480 00000008:00100000:2.0:1789760794.289782:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27480 - object 0x280000401:2 offset 81240064<>81244160 00000100:00100000:2.0:1789760794.289850:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503928960:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.304742:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987141 gen 2 00000008:00010000:3.0:1789760794.304777:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760794.304783:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.304792:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.304852:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760794.304875:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503928960:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.305007:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19834, 19834] 00000080:00200000:2.0:1789760794.305018:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.305023:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.305115:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81244160 i_blocks 158704 00000020:00200000:2.0:1789760794.305119:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.305129:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.305153:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81244160, count: 4096 00000080:00200000:2.0:1789760794.305159:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81244160, bytes: 4096 00000080:00200000:2.0:1789760794.305167:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.305174:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.305176:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81244160, 81248256) 0 0 00000080:00200000:2.0:1789760794.305179:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.305192:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19835, 19835] 00020000:00200000:2.0:1789760794.305198:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81244160 chunk: [81244160, 81248256) 81248256, 4096 00020000:00200000:2.0:1789760794.305202:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.305216:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19835, 19835] 00000008:00010000:2.0:1789760794.305280:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.305286:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.305296:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81244160, 81248256) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.305306:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81244160=0x4d7b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.305349:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.305352:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81248256 00000020:00010000:2.0:1789760794.305359:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.305362:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.305364:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.305368:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.305371:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.305377:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.305383:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.305398:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000008:00100000:2.0:1789760794.305470:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24e00 - object 0x280000401:2 offset 81244160<>81248256 00000100:00100000:3.0:1789760794.305525:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503929088:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.322801:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987142 gen 2 00000008:00010000:3.0:1789760794.322837:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760794.322844:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.322853:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.322873:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760794.322898:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503929088:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.323057:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19835, 19835] 00000080:00200000:2.0:1789760794.323070:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.323075:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.323087:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81248256 i_blocks 158712 00000020:00200000:2.0:1789760794.323090:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.323098:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.323123:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81248256, count: 4096 00000080:00200000:2.0:1789760794.323128:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81248256, bytes: 4096 00000080:00200000:2.0:1789760794.323137:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.323144:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.323146:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81248256, 81252352) 0 0 00000080:00200000:2.0:1789760794.323150:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.323163:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19836, 19836] 00020000:00200000:2.0:1789760794.323170:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81248256 chunk: [81248256, 81252352) 81252352, 4096 00020000:00200000:2.0:1789760794.323174:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.323188:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19836, 19836] 00000008:00010000:2.0:1789760794.323202:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.323207:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.323215:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81248256, 81252352) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.323225:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81248256=0x4d7c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.323271:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.323274:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81252352 00000020:00010000:2.0:1789760794.323280:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.323284:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.323286:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.323290:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.323292:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.323299:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.323305:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.323319:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000008:00100000:2.0:1789760794.323392:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 81248256<>81252352 00000100:00100000:3.0:1789760794.323551:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503929216:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.337103:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987143 gen 2 00000008:00010000:3.0:1789760794.337137:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760794.337144:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.337152:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.337173:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760794.337197:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503929216:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.337199:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19836, 19836] 00000080:00200000:0.0:1789760794.337211:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.337215:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.337235:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81252352 i_blocks 158720 00000020:00200000:0.0:1789760794.337239:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.337246:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.337272:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81252352, count: 4096 00000080:00200000:0.0:1789760794.337277:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81252352, bytes: 4096 00000080:00200000:0.0:1789760794.337286:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.337294:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.337296:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81252352, 81256448) 0 0 00000080:00200000:0.0:1789760794.337299:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.337311:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19837, 19837] 00020000:00200000:0.0:1789760794.337318:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81252352 chunk: [81252352, 81256448) 81256448, 4096 00020000:00200000:0.0:1789760794.337322:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.337346:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19837, 19837] 00000008:00010000:0.0:1789760794.337373:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.337379:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.337388:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81252352, 81256448) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.337399:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81252352=0x4d7d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.337438:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.337441:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81256448 00000020:00010000:0.0:1789760794.337447:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.337449:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.337452:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.337455:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.337457:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.337463:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.337468:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.337480:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760794.337568:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a300 - object 0x280000401:2 offset 81252352<>81256448 00000100:00100000:0.0:1789760794.337611:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760794.337623:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503929344:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.353363:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987144 gen 2 00000008:00010000:0.0:1789760794.353403:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:0.0:1789760794.353410:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.353419:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.353488:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:0.0:1789760794.353515:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503929344:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.353697:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19837, 19837] 00000080:00200000:1.0:1789760794.353710:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.353715:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.353726:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81256448 i_blocks 158728 00000020:00200000:1.0:1789760794.353730:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.353737:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.353762:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81256448, count: 4096 00000080:00200000:1.0:1789760794.353767:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81256448, bytes: 4096 00000080:00200000:1.0:1789760794.353775:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.353782:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.353784:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81256448, 81260544) 0 0 00000080:00200000:1.0:1789760794.353787:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.353798:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19838, 19838] 00020000:00200000:1.0:1789760794.353805:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81256448 chunk: [81256448, 81260544) 81260544, 4096 00020000:00200000:1.0:1789760794.353809:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.353822:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19838, 19838] 00000008:00010000:1.0:1789760794.353834:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.353838:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.353846:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81256448, 81260544) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.353856:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81256448=0x4d7e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.353896:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.353899:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81260544 00000020:00010000:1.0:1789760794.353905:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.353908:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.353912:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.353915:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.353917:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.353924:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.353930:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.353944:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760794.354020:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81256448<>81260544 00000100:00100000:0.0:1789760794.354074:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503929472:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.363743:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987145 gen 2 00000008:00010000:0.0:1789760794.363780:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760794.363787:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.363796:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.363826:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000100:00100000:0.0:1789760794.363854:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503929472:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.363966:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19838, 19838] 00000080:00200000:1.0:1789760794.363981:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.363985:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.363998:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81260544 i_blocks 158736 00000020:00200000:1.0:1789760794.364001:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.364010:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.364060:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81260544, count: 4096 00000080:00200000:1.0:1789760794.364067:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81260544, bytes: 4096 00000080:00200000:1.0:1789760794.364076:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.364099:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.364102:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81260544, 81264640) 0 0 00000080:00200000:1.0:1789760794.364107:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.364122:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19839, 19839] 00020000:00200000:1.0:1789760794.364129:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81260544 chunk: [81260544, 81264640) 81264640, 4096 00020000:00200000:1.0:1789760794.364134:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.364151:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19839, 19839] 00000008:00010000:1.0:1789760794.364165:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.364170:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.364178:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81260544, 81264640) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.364189:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81260544=0x4d7f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.364232:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.364234:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81264640 00000020:00010000:1.0:1789760794.364240:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.364244:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.364247:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.364250:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.364252:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.364259:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.364264:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.364278:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760794.364358:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 81260544<>81264640 00000100:00100000:0.0:1789760794.364603:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503929600:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.377459:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987146 gen 2 00000008:00010000:0.0:1789760794.377494:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760794.377500:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.377509:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.377528:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:0.0:1789760794.377551:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503929600:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.377567:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19839, 19839] 00000080:00200000:1.0:1789760794.377590:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.377594:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.377605:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81264640 i_blocks 158744 00000020:00200000:1.0:1789760794.377617:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.377625:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.377649:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81264640, count: 4096 00000080:00200000:1.0:1789760794.377655:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81264640, bytes: 4096 00000080:00200000:1.0:1789760794.377662:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.377670:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.377671:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81264640, 81268736) 0 0 00000080:00200000:1.0:1789760794.377674:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.377684:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19840, 19840] 00020000:00200000:1.0:1789760794.377690:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81264640 chunk: [81264640, 81268736) 81268736, 4096 00020000:00200000:1.0:1789760794.377693:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.377705:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19840, 19840] 00000008:00010000:1.0:1789760794.377715:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.377718:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.377725:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81264640, 81268736) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.377734:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81264640=0x4d80000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.377767:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.377769:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81268736 00000020:00010000:1.0:1789760794.377774:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.377776:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.377778:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.377780:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.377782:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.377787:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.377791:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.377801:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760794.377860:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 81264640<>81268736 00000100:00100000:0.0:1789760794.377906:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760794.378012:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503929728:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.393852:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987147 gen 2 00000008:00010000:0.0:1789760794.393891:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760794.393897:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.393907:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.394024:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760794.394064:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503929728:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.394154:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19840, 19840] 00000080:00200000:1.0:1789760794.394165:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.394170:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.394183:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81268736 i_blocks 158752 00000020:00200000:1.0:1789760794.394186:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.394194:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.394219:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81268736, count: 4096 00000080:00200000:1.0:1789760794.394225:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81268736, bytes: 4096 00000080:00200000:1.0:1789760794.394233:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.394241:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.394243:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81268736, 81272832) 0 0 00000080:00200000:1.0:1789760794.394246:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.394259:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19841, 19841] 00020000:00200000:1.0:1789760794.394266:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81268736 chunk: [81268736, 81272832) 81272832, 4096 00020000:00200000:1.0:1789760794.394269:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.394284:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19841, 19841] 00000008:00010000:1.0:1789760794.394297:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.394302:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.394310:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81268736, 81272832) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.394322:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81268736=0x4d81000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.394364:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.394369:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81272832 00000020:00010000:1.0:1789760794.394376:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.394379:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.394381:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.394385:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.394387:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.394393:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.394399:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.394412:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760794.394486:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 81268736<>81272832 00000100:00100000:0.0:1789760794.394714:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503929856:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.403355:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987148 gen 2 00000008:00010000:0.0:1789760794.403391:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:0.0:1789760794.403397:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.403407:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.403495:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:0.0:1789760794.403520:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503929856:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.403793:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19841, 19841] 00000080:00200000:1.0:1789760794.403805:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.403810:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.403822:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81272832 i_blocks 158760 00000020:00200000:1.0:1789760794.403825:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.403834:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.403858:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81272832, count: 4096 00000080:00200000:1.0:1789760794.403864:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81272832, bytes: 4096 00000080:00200000:1.0:1789760794.403871:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.403878:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.403880:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81272832, 81276928) 0 0 00000080:00200000:1.0:1789760794.403883:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.403896:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19842, 19842] 00020000:00200000:1.0:1789760794.403903:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81272832 chunk: [81272832, 81276928) 81276928, 4096 00020000:00200000:1.0:1789760794.403906:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.403921:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19842, 19842] 00000008:00010000:1.0:1789760794.403936:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.403940:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.403950:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81272832, 81276928) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.403961:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81272832=0x4d82000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.404012:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.404014:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81276928 00000020:00010000:1.0:1789760794.404021:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.404023:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.404026:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.404029:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.404316:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.404327:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.404333:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.404347:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58780 00000008:00100000:1.0:1789760794.404423:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 81272832<>81276928 00000100:00100000:0.0:1789760794.404558:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503929984:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.415551:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987149 gen 2 00000008:00010000:0.0:1789760794.415606:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58780 00000020:00200000:0.0:1789760794.415612:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.415620:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.415689:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58780 00000100:00100000:0.0:1789760794.415712:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503929984:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.415885:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19842, 19842] 00000080:00200000:1.0:1789760794.415907:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.415913:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.415925:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81276928 i_blocks 158768 00000020:00200000:1.0:1789760794.415929:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.415937:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.415962:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81276928, count: 4096 00000080:00200000:1.0:1789760794.415967:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81276928, bytes: 4096 00000080:00200000:1.0:1789760794.415974:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.415982:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.415984:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81276928, 81281024) 0 0 00000080:00200000:1.0:1789760794.415988:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.415999:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19843, 19843] 00020000:00200000:1.0:1789760794.416006:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81276928 chunk: [81276928, 81281024) 81281024, 4096 00020000:00200000:1.0:1789760794.416010:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.416023:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19843, 19843] 00000008:00010000:1.0:1789760794.416063:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.416068:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.416077:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81276928, 81281024) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.416087:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81276928=0x4d83000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.416141:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.416144:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81281024 00000020:00010000:1.0:1789760794.416151:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.416154:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.416157:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.416160:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.416162:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.416168:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.416174:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.416188:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760794.416276:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 81276928<>81281024 00000100:00100000:0.0:1789760794.416339:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503930112:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.431086:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987150 gen 2 00000008:00010000:0.0:1789760794.431124:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:0.0:1789760794.431131:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.431140:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.431223:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000100:00100000:0.0:1789760794.431248:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503930112:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.432230:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19843, 19843] 00000080:00200000:1.0:1789760794.432245:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.432251:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.432264:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81281024 i_blocks 158776 00000020:00200000:1.0:1789760794.432268:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.432277:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.432302:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81281024, count: 4096 00000080:00200000:1.0:1789760794.432308:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81281024, bytes: 4096 00000080:00200000:1.0:1789760794.432317:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.432325:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.432327:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81281024, 81285120) 0 0 00000080:00200000:1.0:1789760794.432331:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.432343:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19844, 19844] 00020000:00200000:1.0:1789760794.432349:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81281024 chunk: [81281024, 81285120) 81285120, 4096 00020000:00200000:1.0:1789760794.432353:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.432368:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19844, 19844] 00000008:00010000:1.0:1789760794.432382:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.432386:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.432395:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81281024, 81285120) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.432409:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81281024=0x4d84000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.432458:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.432461:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81285120 00000020:00010000:1.0:1789760794.432467:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.432470:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.432473:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.432476:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.432479:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.432485:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.432490:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.432507:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760794.432593:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 81281024<>81285120 00000100:00100000:0.0:1789760794.432781:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503930240:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.442444:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987151 gen 2 00000008:00010000:0.0:1789760794.442490:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58000 00000020:00200000:0.0:1789760794.442496:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.442505:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.442525:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58000 00000100:00100000:0.0:1789760794.442548:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503930240:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.442560:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19844, 19844] 00000080:00200000:2.0:1789760794.442584:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.442590:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.442603:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81285120 i_blocks 158784 00000020:00200000:2.0:1789760794.442607:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.442616:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.442640:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81285120, count: 4096 00000080:00200000:2.0:1789760794.442646:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81285120, bytes: 4096 00000080:00200000:2.0:1789760794.442654:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.442662:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.442664:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81285120, 81289216) 0 0 00000080:00200000:2.0:1789760794.442667:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.442680:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19845, 19845] 00020000:00200000:2.0:1789760794.442687:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81285120 chunk: [81285120, 81289216) 81289216, 4096 00020000:00200000:2.0:1789760794.442691:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.442719:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19845, 19845] 00000008:00010000:2.0:1789760794.442735:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.442740:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.442748:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81285120, 81289216) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.442760:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81285120=0x4d85000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.442810:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.442813:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81289216 00000020:00010000:2.0:1789760794.442820:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.442823:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.442826:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.442830:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.442832:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.442840:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.442850:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.442865:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c300 00000008:00100000:2.0:1789760794.442953:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24700 - object 0x280000401:2 offset 81285120<>81289216 00000100:00100000:3.0:1789760794.443223:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760794.443239:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503930368:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.459664:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987152 gen 2 00000008:00010000:2.0:1789760794.459820:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:2.0:1789760794.459828:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760794.459836:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760794.459852:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:2.0:1789760794.459872:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503930368:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.460000:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19845, 19845] 00000080:00200000:0.0:1789760794.460012:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.460016:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.460026:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81289216 i_blocks 158792 00000020:00200000:0.0:1789760794.460028:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.460628:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.460652:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81289216, count: 4096 00000080:00200000:0.0:1789760794.460659:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81289216, bytes: 4096 00000080:00200000:0.0:1789760794.460667:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.460675:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.460676:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81289216, 81293312) 0 0 00000080:00200000:0.0:1789760794.460680:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.460691:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19846, 19846] 00020000:00200000:0.0:1789760794.460699:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81289216 chunk: [81289216, 81293312) 81293312, 4096 00020000:00200000:0.0:1789760794.460702:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.460717:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19846, 19846] 00000008:00010000:0.0:1789760794.460731:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.460735:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.460744:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81289216, 81293312) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.460754:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81289216=0x4d86000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.460792:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.460794:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81293312 00000020:00010000:0.0:1789760794.460801:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.460804:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.460807:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.460810:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.460813:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.460820:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.460826:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.460913:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:2.0:1789760794.460996:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24700 - object 0x280000401:2 offset 81289216<>81293312 00000100:00100000:3.0:1789760794.461054:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760794.461308:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503930496:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.474458:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987153 gen 2 00000008:00010000:3.0:1789760794.474492:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:3.0:1789760794.474498:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.474507:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.474525:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:3.0:1789760794.474548:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503930496:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.474652:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19846, 19846] 00000080:00200000:0.0:1789760794.474665:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.474669:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.474681:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81293312 i_blocks 158800 00000020:00200000:0.0:1789760794.474684:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.474692:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.474717:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81293312, count: 4096 00000080:00200000:0.0:1789760794.474722:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81293312, bytes: 4096 00000080:00200000:0.0:1789760794.474730:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.474737:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.474739:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81293312, 81297408) 0 0 00000080:00200000:0.0:1789760794.474742:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.474754:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19847, 19847] 00020000:00200000:0.0:1789760794.474761:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81293312 chunk: [81293312, 81297408) 81297408, 4096 00020000:00200000:0.0:1789760794.474765:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.474779:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19847, 19847] 00000008:00010000:0.0:1789760794.474791:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.474797:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.474805:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81293312, 81297408) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.474815:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81293312=0x4d87000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.474856:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.474858:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81297408 00000020:00010000:0.0:1789760794.474864:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.474867:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.474870:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.474873:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.474876:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.474883:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.474899:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.474913:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760794.474992:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a300 - object 0x280000401:2 offset 81293312<>81297408 00000100:00100000:0.0:1789760794.475028:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760794.475073:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503930624:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.484303:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987154 gen 2 00000008:00010000:0.0:1789760794.484336:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:0.0:1789760794.484343:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.484353:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.484412:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:0.0:1789760794.484438:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503930624:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.487539:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19847, 19847] 00000080:00200000:2.0:1789760794.487558:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.487564:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.487589:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81297408 i_blocks 158808 00000020:00200000:2.0:1789760794.487594:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.487606:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.487632:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81297408, count: 4096 00000080:00200000:2.0:1789760794.487639:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81297408, bytes: 4096 00000080:00200000:2.0:1789760794.487650:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.487658:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.487660:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81297408, 81301504) 0 0 00000080:00200000:2.0:1789760794.487765:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.487780:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19848, 19848] 00020000:00200000:2.0:1789760794.487787:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81297408 chunk: [81297408, 81301504) 81301504, 4096 00020000:00200000:2.0:1789760794.487791:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.487808:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19848, 19848] 00000008:00010000:2.0:1789760794.487823:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.487828:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.487907:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81297408, 81301504) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.487920:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81297408=0x4d88000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.487977:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.487980:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81301504 00000020:00010000:2.0:1789760794.487986:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.487989:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.487992:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.487995:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.487998:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.488005:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.488011:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.488286:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c300 00000008:00100000:2.0:1789760794.488449:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24380 - object 0x280000401:2 offset 81297408<>81301504 00000100:00100000:3.0:1789760794.488513:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503930752:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.506433:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987155 gen 2 00000008:00010000:3.0:1789760794.506470:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760794.506477:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.506488:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.506556:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760794.506588:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503930752:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.509737:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19848, 19848] 00000080:00200000:0.0:1789760794.509757:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.509764:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.509779:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81301504 i_blocks 158816 00000020:00200000:0.0:1789760794.509784:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.509797:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.509822:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81301504, count: 4096 00000080:00200000:0.0:1789760794.509829:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81301504, bytes: 4096 00000080:00200000:0.0:1789760794.509840:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.509848:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.509849:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81301504, 81305600) 0 0 00000080:00200000:0.0:1789760794.509854:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.509866:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19849, 19849] 00020000:00200000:0.0:1789760794.509873:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81301504 chunk: [81301504, 81305600) 81305600, 4096 00020000:00200000:0.0:1789760794.509877:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.509892:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19849, 19849] 00000008:00010000:0.0:1789760794.509907:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.509912:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.509920:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81301504, 81305600) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.509933:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81301504=0x4d89000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.509984:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.509987:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81305600 00000020:00010000:0.0:1789760794.509993:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.509996:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.509998:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.510001:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.510004:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.510012:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.510017:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.510688:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760794.510771:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a300 - object 0x280000401:2 offset 81301504<>81305600 00000100:00100000:1.0:1789760794.510828:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760794.513709:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503930880:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.524092:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987156 gen 2 00000008:00010000:1.0:1789760794.524131:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760794.524138:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.524147:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.524210:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760794.524237:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503930880:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760794.526445:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19849, 19849] 00000080:00200000:3.0:1789760794.526463:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.526469:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.526483:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81305600 i_blocks 158824 00000020:00200000:3.0:1789760794.526488:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.526500:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.526525:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81305600, count: 4096 00000080:00200000:3.0:1789760794.526532:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81305600, bytes: 4096 00000080:00200000:3.0:1789760794.526543:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.526550:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760794.526552:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81305600, 81309696) 0 0 00000080:00200000:3.0:1789760794.526556:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.526580:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19850, 19850] 00020000:00200000:3.0:1789760794.526587:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81305600 chunk: [81305600, 81309696) 81309696, 4096 00020000:00200000:3.0:1789760794.526591:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.526616:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19850, 19850] 00000008:00010000:3.0:1789760794.526629:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.526635:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760794.526644:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81305600, 81309696) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.526655:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81305600=0x4d8a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.526708:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.526711:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81309696 00000020:00010000:3.0:1789760794.526718:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.526721:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.526724:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.526727:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.526729:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.526736:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.526742:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.526764:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1180 00000008:00100000:3.0:1789760794.526844:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 81305600<>81309696 00000100:00100000:3.0:1789760794.526890:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760794.526908:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503931008:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.539896:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987157 gen 2 00000008:00010000:3.0:1789760794.539932:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1180 00000020:00200000:3.0:1789760794.539938:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.539946:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.539970:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1180 00000100:00100000:3.0:1789760794.539996:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503931008:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.541640:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19850, 19850] 00000080:00200000:1.0:1789760794.541657:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.541662:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.541676:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81309696 i_blocks 158832 00000020:00200000:1.0:1789760794.541680:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.541691:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.541714:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81309696, count: 4096 00000080:00200000:1.0:1789760794.541721:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81309696, bytes: 4096 00000080:00200000:1.0:1789760794.541730:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.541738:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.541740:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81309696, 81313792) 0 0 00000080:00200000:1.0:1789760794.541744:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.541756:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19851, 19851] 00020000:00200000:1.0:1789760794.541763:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81309696 chunk: [81309696, 81313792) 81313792, 4096 00020000:00200000:1.0:1789760794.541767:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.541781:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19851, 19851] 00000008:00010000:1.0:1789760794.541795:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.541800:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.541809:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81309696, 81313792) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.541820:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81309696=0x4d8b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.541870:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.541872:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81313792 00000020:00010000:1.0:1789760794.541879:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.541882:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.541884:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.541887:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.541890:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.541896:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.541902:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.541920:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760794.541997:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81309696<>81313792 00000100:00100000:0.0:1789760794.542131:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760794.542150:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503931136:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.549538:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987158 gen 2 00000008:00010000:0.0:1789760794.549583:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:0.0:1789760794.549590:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.549599:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.549624:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:0.0:1789760794.549648:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503931136:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.549825:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19851, 19851] 00000080:00200000:1.0:1789760794.549838:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.549842:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.549854:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81313792 i_blocks 158840 00000020:00200000:1.0:1789760794.549857:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.549865:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.549890:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81313792, count: 4096 00000080:00200000:1.0:1789760794.549896:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81313792, bytes: 4096 00000080:00200000:1.0:1789760794.549903:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.549910:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.549912:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81313792, 81317888) 0 0 00000080:00200000:1.0:1789760794.549916:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.549928:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19852, 19852] 00020000:00200000:1.0:1789760794.549935:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81313792 chunk: [81313792, 81317888) 81317888, 4096 00020000:00200000:1.0:1789760794.549939:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.549955:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19852, 19852] 00000008:00010000:1.0:1789760794.549970:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.549975:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.549983:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81313792, 81317888) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.549994:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81313792=0x4d8c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.550071:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.550074:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81317888 00000020:00010000:1.0:1789760794.550081:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.550084:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.550087:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.550090:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.550092:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.550098:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.550103:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.550115:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760794.550187:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81313792<>81317888 00000100:00100000:0.0:1789760794.550253:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760794.550267:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503931264:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.567450:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987159 gen 2 00000008:00010000:0.0:1789760794.567488:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d586c0 00000020:00200000:0.0:1789760794.567494:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.567504:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.567527:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d586c0 00000100:00100000:0.0:1789760794.567551:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503931264:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.567630:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19852, 19852] 00000080:00200000:1.0:1789760794.567642:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.567647:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.567659:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81317888 i_blocks 158848 00000020:00200000:1.0:1789760794.567663:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.567671:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.567697:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81317888, count: 4096 00000080:00200000:1.0:1789760794.567703:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81317888, bytes: 4096 00000080:00200000:1.0:1789760794.567712:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.567719:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.567721:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81317888, 81321984) 0 0 00000080:00200000:1.0:1789760794.567725:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.567736:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19853, 19853] 00020000:00200000:1.0:1789760794.567742:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81317888 chunk: [81317888, 81321984) 81321984, 4096 00020000:00200000:1.0:1789760794.567746:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.567760:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19853, 19853] 00000008:00010000:1.0:1789760794.567773:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.567778:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.567787:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81317888, 81321984) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.567798:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81317888=0x4d8d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.567843:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.567845:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81321984 00000020:00010000:1.0:1789760794.567852:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.567855:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.567857:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.567860:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.567862:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.567868:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.567874:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.567988:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760794.568172:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 81317888<>81321984 00000100:00100000:0.0:1789760794.568219:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503931392:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.584942:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987160 gen 2 00000008:00010000:0.0:1789760794.584979:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58a80 00000020:00200000:0.0:1789760794.584985:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.584994:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.585018:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58a80 00000100:00100000:0.0:1789760794.585555:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503931392:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760794.585725:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19853, 19853] 00000080:00200000:3.0:1789760794.585737:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.585742:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.585753:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81321984 i_blocks 158856 00000020:00200000:3.0:1789760794.585756:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.585764:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.585789:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81321984, count: 4096 00000080:00200000:3.0:1789760794.585794:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81321984, bytes: 4096 00000080:00200000:3.0:1789760794.585802:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.585809:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760794.585812:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81321984, 81326080) 0 0 00000080:00200000:3.0:1789760794.585815:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.585826:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19854, 19854] 00020000:00200000:3.0:1789760794.585833:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81321984 chunk: [81321984, 81326080) 81326080, 4096 00020000:00200000:3.0:1789760794.585837:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.585849:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19854, 19854] 00000008:00010000:3.0:1789760794.586173:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.586180:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760794.586190:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81321984, 81326080) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.586202:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81321984=0x4d8e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.586245:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.586247:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81326080 00000020:00010000:3.0:1789760794.586254:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.586257:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.586259:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.586262:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.586265:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.586270:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.586276:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.586291:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1180 00000008:00100000:3.0:1789760794.586930:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 81321984<>81326080 00000100:00100000:3.0:1789760794.586979:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503931520:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.605513:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987161 gen 2 00000008:00010000:3.0:1789760794.605548:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1180 00000020:00200000:3.0:1789760794.605553:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.605562:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.605643:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1180 00000100:00100000:3.0:1789760794.605666:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503931520:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.607422:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19854, 19854] 00000080:00200000:2.0:1789760794.607439:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.607445:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.607459:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81326080 i_blocks 158864 00000020:00200000:2.0:1789760794.607463:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.607473:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.607497:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81326080, count: 4096 00000080:00200000:2.0:1789760794.607503:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81326080, bytes: 4096 00000080:00200000:2.0:1789760794.607514:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.607521:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.607523:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81326080, 81330176) 0 0 00000080:00200000:2.0:1789760794.607527:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.607539:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19855, 19855] 00020000:00200000:2.0:1789760794.607546:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81326080 chunk: [81326080, 81330176) 81330176, 4096 00020000:00200000:2.0:1789760794.607550:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.607565:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19855, 19855] 00000008:00010000:2.0:1789760794.607588:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.607592:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.607600:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81326080, 81330176) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.607934:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81326080=0x4d8f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.607990:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.607992:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81330176 00000020:00010000:2.0:1789760794.607999:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.608002:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.608005:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.608008:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.608011:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.608018:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.608023:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.609693:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cc00 00000008:00100000:2.0:1789760794.609794:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25c00 - object 0x280000401:2 offset 81326080<>81330176 00000100:00100000:2.0:1789760794.609864:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503931648:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.625661:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987162 gen 2 00000008:00010000:3.0:1789760794.625699:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760794.625706:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.625716:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.625782:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760794.625808:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503931648:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.625879:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19855, 19855] 00000080:00200000:0.0:1789760794.625892:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.625897:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.625909:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81330176 i_blocks 158872 00000020:00200000:0.0:1789760794.625913:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.625921:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.625949:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81330176, count: 4096 00000080:00200000:0.0:1789760794.625955:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81330176, bytes: 4096 00000080:00200000:0.0:1789760794.625963:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.625970:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.625972:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81330176, 81334272) 0 0 00000080:00200000:0.0:1789760794.625976:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.625987:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19856, 19856] 00020000:00200000:0.0:1789760794.625994:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81330176 chunk: [81330176, 81334272) 81334272, 4096 00020000:00200000:0.0:1789760794.625999:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.626012:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19856, 19856] 00000008:00010000:0.0:1789760794.626025:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.626030:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.626056:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81330176, 81334272) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.626068:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81330176=0x4d90000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.626110:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.626113:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81334272 00000020:00010000:0.0:1789760794.626120:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.626123:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.626126:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.626129:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.626131:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.626138:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.626144:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.626158:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760794.626238:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99c00 - object 0x280000401:2 offset 81330176<>81334272 00000100:00100000:1.0:1789760794.626316:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503931776:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.640083:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987163 gen 2 00000008:00010000:1.0:1789760794.640118:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760794.640192:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.640202:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.640259:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760794.640283:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503931776:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.640350:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19856, 19856] 00000080:00200000:0.0:1789760794.640362:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.640366:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.640379:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81334272 i_blocks 158880 00000020:00200000:0.0:1789760794.640382:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.640389:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.640412:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81334272, count: 4096 00000080:00200000:0.0:1789760794.640417:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81334272, bytes: 4096 00000080:00200000:0.0:1789760794.640426:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.640433:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.640435:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81334272, 81338368) 0 0 00000080:00200000:0.0:1789760794.640439:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.640450:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19857, 19857] 00020000:00200000:0.0:1789760794.640456:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81334272 chunk: [81334272, 81338368) 81338368, 4096 00020000:00200000:0.0:1789760794.640461:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.640473:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19857, 19857] 00000008:00010000:0.0:1789760794.640540:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.640546:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.640554:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81334272, 81338368) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.640564:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81334272=0x4d91000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.640791:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.640858:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81338368 00000020:00010000:0.0:1789760794.640866:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.640869:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.640872:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.640875:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.640877:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.640883:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.640951:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.640967:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760794.641103:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98380 - object 0x280000401:2 offset 81334272<>81338368 00000100:00100000:1.0:1789760794.641158:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503931904:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.665192:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987164 gen 2 00000008:00010000:1.0:1789760794.665231:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760794.665238:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.665247:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.665271:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760794.665296:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503931904:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.665422:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19857, 19857] 00000080:00200000:0.0:1789760794.665436:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.665441:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.665453:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81338368 i_blocks 158888 00000020:00200000:0.0:1789760794.665456:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.665464:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.665488:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81338368, count: 4096 00000080:00200000:0.0:1789760794.665494:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81338368, bytes: 4096 00000080:00200000:0.0:1789760794.665501:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.665508:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.665510:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81338368, 81342464) 0 0 00000080:00200000:0.0:1789760794.665514:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.665527:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19858, 19858] 00020000:00200000:0.0:1789760794.665534:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81338368 chunk: [81338368, 81342464) 81342464, 4096 00020000:00200000:0.0:1789760794.665539:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.665554:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19858, 19858] 00000008:00010000:0.0:1789760794.665583:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.665589:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.665597:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81338368, 81342464) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.665616:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81338368=0x4d92000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.665663:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.665666:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81342464 00000020:00010000:0.0:1789760794.665673:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.665676:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.665679:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.665682:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.665685:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.665692:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.665698:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.665712:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760794.665786:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9ad80 - object 0x280000401:2 offset 81338368<>81342464 00000100:00100000:1.0:1789760794.666017:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503932032:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.683103:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987165 gen 2 00000008:00010000:1.0:1789760794.683143:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760794.683150:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.683159:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.683181:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760794.683209:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503932032:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.683374:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19858, 19858] 00000080:00200000:0.0:1789760794.683386:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.683391:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.683405:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81342464 i_blocks 158896 00000020:00200000:0.0:1789760794.683408:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.683416:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.683442:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81342464, count: 4096 00000080:00200000:0.0:1789760794.683447:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81342464, bytes: 4096 00000080:00200000:0.0:1789760794.683455:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.683462:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.683464:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81342464, 81346560) 0 0 00000080:00200000:0.0:1789760794.683468:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.683481:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19859, 19859] 00020000:00200000:0.0:1789760794.683488:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81342464 chunk: [81342464, 81346560) 81346560, 4096 00020000:00200000:0.0:1789760794.683493:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.683507:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19859, 19859] 00000008:00010000:0.0:1789760794.683520:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.683525:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.683533:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81342464, 81346560) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.683545:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81342464=0x4d93000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.683598:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.683601:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81346560 00000020:00010000:0.0:1789760794.683618:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.683621:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.683624:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.683627:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.683630:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.683638:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.683643:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.683657:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760794.683731:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b100 - object 0x280000401:2 offset 81342464<>81346560 00000100:00100000:1.0:1789760794.683936:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503932160:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.696826:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987166 gen 2 00000008:00010000:1.0:1789760794.696865:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760794.696872:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.696881:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.696901:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760794.696926:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503932160:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.697091:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19859, 19859] 00000080:00200000:0.0:1789760794.697103:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.697107:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.697120:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81346560 i_blocks 158904 00000020:00200000:0.0:1789760794.697123:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.697131:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.697156:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81346560, count: 4096 00000080:00200000:0.0:1789760794.697162:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81346560, bytes: 4096 00000080:00200000:0.0:1789760794.697169:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.697176:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.697178:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81346560, 81350656) 0 0 00000080:00200000:0.0:1789760794.697182:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.697194:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19860, 19860] 00020000:00200000:0.0:1789760794.697200:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81346560 chunk: [81346560, 81350656) 81350656, 4096 00020000:00200000:0.0:1789760794.697205:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.697220:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19860, 19860] 00000008:00010000:0.0:1789760794.697233:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.697238:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.697247:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81346560, 81350656) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.697258:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81346560=0x4d94000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.697304:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.697306:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81350656 00000020:00010000:0.0:1789760794.697313:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.697316:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.697318:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.697322:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.697324:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.697330:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.697336:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.697360:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760794.697437:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9aa00 - object 0x280000401:2 offset 81346560<>81350656 00000100:00100000:1.0:1789760794.697691:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503932288:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.717134:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987167 gen 2 00000008:00010000:1.0:1789760794.717286:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760794.717293:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.717302:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.717379:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760794.717407:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503932288:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.718562:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19860, 19860] 00000080:00200000:0.0:1789760794.718586:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.718591:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.718754:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81350656 i_blocks 158912 00000020:00200000:0.0:1789760794.718760:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.718770:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.718796:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81350656, count: 4096 00000080:00200000:0.0:1789760794.718802:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81350656, bytes: 4096 00000080:00200000:0.0:1789760794.718812:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.718819:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.718821:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81350656, 81354752) 0 0 00000080:00200000:0.0:1789760794.718825:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.718836:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19861, 19861] 00020000:00200000:0.0:1789760794.718843:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81350656 chunk: [81350656, 81354752) 81354752, 4096 00020000:00200000:0.0:1789760794.718848:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.718862:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19861, 19861] 00000008:00010000:0.0:1789760794.718876:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.718881:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.718890:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81350656, 81354752) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.718900:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81350656=0x4d95000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.719327:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.719330:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81354752 00000020:00010000:0.0:1789760794.719338:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.719341:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.719344:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.719347:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.719349:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.719357:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.719363:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.719381:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760794.719460:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b800 - object 0x280000401:2 offset 81350656<>81354752 00000100:00100000:1.0:1789760794.719521:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503932416:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.735332:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987168 gen 2 00000008:00010000:1.0:1789760794.735367:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760794.735375:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.735383:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.735447:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760794.735472:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503932416:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760794.736972:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19861, 19861] 00000080:00200000:3.0:1789760794.736986:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.736992:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.737245:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81354752 i_blocks 158920 00000020:00200000:3.0:1789760794.737250:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.737262:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.737288:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81354752, count: 4096 00000080:00200000:3.0:1789760794.737295:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81354752, bytes: 4096 00000080:00200000:3.0:1789760794.737305:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.737312:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760794.737314:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81354752, 81358848) 0 0 00000080:00200000:3.0:1789760794.737317:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.737330:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19862, 19862] 00020000:00200000:3.0:1789760794.737337:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81354752 chunk: [81354752, 81358848) 81358848, 4096 00020000:00200000:3.0:1789760794.737341:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.737356:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19862, 19862] 00000008:00010000:3.0:1789760794.737370:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.737374:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760794.737383:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81354752, 81358848) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.737394:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81354752=0x4d96000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.737444:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.737447:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81358848 00000020:00010000:3.0:1789760794.737453:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.737456:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.737459:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.737462:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.737465:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.737472:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.737477:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.737498:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1180 00000008:00100000:3.0:1789760794.739317:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070ce00 - object 0x280000401:2 offset 81354752<>81358848 00000100:00100000:3.0:1789760794.739382:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760794.739402:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503932544:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.762072:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987169 gen 2 00000008:00010000:2.0:1789760794.762322:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1180 00000020:00200000:2.0:1789760794.762329:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760794.762338:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760794.762396:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1180 00000100:00100000:2.0:1789760794.762421:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503932544:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760794.762495:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19862, 19862] 00000080:00200000:3.0:1789760794.762506:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.762512:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.762700:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81358848 i_blocks 158928 00000020:00200000:2.0:1789760794.762705:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.762713:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.762741:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81358848, count: 4096 00000080:00200000:2.0:1789760794.762748:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81358848, bytes: 4096 00000080:00200000:2.0:1789760794.762756:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.762764:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.762766:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81358848, 81362944) 0 0 00000080:00200000:2.0:1789760794.762770:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.762782:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19863, 19863] 00020000:00200000:2.0:1789760794.762790:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81358848 chunk: [81358848, 81362944) 81362944, 4096 00020000:00200000:2.0:1789760794.762793:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.762809:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19863, 19863] 00000008:00010000:2.0:1789760794.762823:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.762828:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.762840:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81358848, 81362944) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.762850:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81358848=0x4d97000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.764700:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.764709:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81362944 00000020:00010000:2.0:1789760794.764722:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.764726:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.764731:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.764735:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.764738:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.764749:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.764759:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.764785:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cf00 00000008:00100000:2.0:1789760794.764889:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25180 - object 0x280000401:2 offset 81358848<>81362944 00000100:00100000:3.0:1789760794.764985:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503932672:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.776368:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987170 gen 2 00000008:00010000:3.0:1789760794.776404:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760794.776410:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.776419:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.776438:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760794.776461:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503932672:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.776546:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19863, 19863] 00000080:00200000:2.0:1789760794.776557:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.776562:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.776582:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81362944 i_blocks 158936 00000020:00200000:2.0:1789760794.776586:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.776593:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.776680:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81362944, count: 4096 00000080:00200000:2.0:1789760794.776686:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81362944, bytes: 4096 00000080:00200000:2.0:1789760794.776694:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.776701:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.776703:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81362944, 81367040) 0 0 00000080:00200000:2.0:1789760794.776707:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.776721:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19864, 19864] 00020000:00200000:2.0:1789760794.776728:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81362944 chunk: [81362944, 81367040) 81367040, 4096 00020000:00200000:2.0:1789760794.776731:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.776746:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19864, 19864] 00000008:00010000:2.0:1789760794.776760:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.776816:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.776825:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81362944, 81367040) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.776836:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81362944=0x4d98000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.777533:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.777537:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81367040 00000020:00010000:2.0:1789760794.777545:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.777548:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.777551:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.777554:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.777557:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.777573:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.777580:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.777598:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c000 00000008:00100000:2.0:1789760794.777956:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27800 - object 0x280000401:2 offset 81362944<>81367040 00000100:00100000:3.0:1789760794.778935:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760794.778957:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503932800:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.799693:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987171 gen 2 00000008:00010000:3.0:1789760794.799728:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760794.799735:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.799743:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.799802:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760794.799828:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503932800:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760794.802518:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19864, 19864] 00000080:00200000:2.0:1789760794.802538:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.802543:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.802589:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81367040 i_blocks 158944 00000020:00200000:2.0:1789760794.802594:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.802664:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.802691:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81367040, count: 4096 00000080:00200000:2.0:1789760794.802697:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81367040, bytes: 4096 00000080:00200000:2.0:1789760794.802708:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.802714:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760794.802717:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81367040, 81371136) 0 0 00000080:00200000:2.0:1789760794.802721:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.802734:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19865, 19865] 00020000:00200000:2.0:1789760794.802740:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81367040 chunk: [81367040, 81371136) 81371136, 4096 00020000:00200000:2.0:1789760794.802745:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.802760:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19865, 19865] 00000008:00010000:2.0:1789760794.802774:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.802778:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760794.802787:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81367040, 81371136) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.802797:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81367040=0x4d99000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.802893:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.802896:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81371136 00000020:00010000:2.0:1789760794.802903:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.802906:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.802909:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.802912:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.802922:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.802930:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.802935:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.802948:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ce40 00000008:00100000:2.0:1789760794.803806:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26a00 - object 0x280000401:2 offset 81367040<>81371136 00000100:00100000:2.0:1789760794.803865:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503932928:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.819332:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987172 gen 2 00000008:00010000:3.0:1789760794.819370:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760794.819376:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.819385:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.819406:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760794.819432:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503932928:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.819718:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19865, 19865] 00000080:00200000:1.0:1789760794.819730:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.819735:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.819748:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81371136 i_blocks 158952 00000020:00200000:1.0:1789760794.819751:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.819759:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.819785:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81371136, count: 4096 00000080:00200000:1.0:1789760794.819792:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81371136, bytes: 4096 00000080:00200000:1.0:1789760794.819801:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.819808:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.819810:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81371136, 81375232) 0 0 00000080:00200000:1.0:1789760794.819814:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.819828:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19866, 19866] 00020000:00200000:1.0:1789760794.819835:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81371136 chunk: [81371136, 81375232) 81375232, 4096 00020000:00200000:1.0:1789760794.819840:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.819855:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19866, 19866] 00000008:00010000:1.0:1789760794.819867:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.819872:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.819880:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81371136, 81375232) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.819890:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81371136=0x4d9a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.819936:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.819939:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81375232 00000020:00010000:1.0:1789760794.819945:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.819948:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.819950:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.819953:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.819956:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.819962:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.819967:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.819981:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760794.820253:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 81371136<>81375232 00000100:00100000:0.0:1789760794.820316:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503933056:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.839362:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987173 gen 2 00000008:00010000:0.0:1789760794.839404:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58540 00000020:00200000:0.0:1789760794.839412:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.839421:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.839500:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58540 00000100:00100000:0.0:1789760794.839525:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503933056:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.842798:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19866, 19866] 00000080:00200000:1.0:1789760794.842818:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.842823:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.842834:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81375232 i_blocks 158960 00000020:00200000:1.0:1789760794.842895:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.842908:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.842934:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81375232, count: 4096 00000080:00200000:1.0:1789760794.842942:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81375232, bytes: 4096 00000080:00200000:1.0:1789760794.842954:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.842962:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.842964:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81375232, 81379328) 0 0 00000080:00200000:1.0:1789760794.842969:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.842982:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19867, 19867] 00020000:00200000:1.0:1789760794.842990:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81375232 chunk: [81375232, 81379328) 81379328, 4096 00020000:00200000:1.0:1789760794.842994:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.843011:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19867, 19867] 00000008:00010000:1.0:1789760794.843026:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.843031:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.843060:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81375232, 81379328) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.843073:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81375232=0x4d9b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.843124:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.843127:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81379328 00000020:00010000:1.0:1789760794.843134:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.843137:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.843139:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.843142:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.843144:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.843150:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.843155:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.843308:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760794.843398:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 81375232<>81379328 00000100:00100000:0.0:1789760794.843691:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503933184:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.861851:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987174 gen 2 00000008:00010000:0.0:1789760794.861888:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d583c0 00000020:00200000:0.0:1789760794.861895:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.861906:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.861926:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d583c0 00000100:00100000:0.0:1789760794.861949:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503933184:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.862750:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19867, 19867] 00000080:00200000:1.0:1789760794.862766:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.862771:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.862784:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81379328 i_blocks 158968 00000020:00200000:1.0:1789760794.862788:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.862796:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.862823:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81379328, count: 4096 00000080:00200000:1.0:1789760794.862830:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81379328, bytes: 4096 00000080:00200000:1.0:1789760794.862839:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.862846:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.862847:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81379328, 81383424) 0 0 00000080:00200000:1.0:1789760794.862851:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.862863:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19868, 19868] 00020000:00200000:1.0:1789760794.862870:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81379328 chunk: [81379328, 81383424) 81383424, 4096 00020000:00200000:1.0:1789760794.862874:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.862889:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19868, 19868] 00000008:00010000:1.0:1789760794.862903:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.862907:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.862917:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81379328, 81383424) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.862928:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81379328=0x4d9c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.862974:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.862977:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81383424 00000020:00010000:1.0:1789760794.862982:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.862985:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.862988:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.862991:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.862993:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.862999:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.863005:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.863021:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760794.863139:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 81379328<>81383424 00000100:00100000:0.0:1789760794.863305:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760794.863323:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503933312:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.873941:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987175 gen 2 00000008:00010000:0.0:1789760794.873981:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:0.0:1789760794.873987:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.873996:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.874016:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:0.0:1789760794.874072:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503933312:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.874169:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19868, 19868] 00000080:00200000:1.0:1789760794.874180:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.874183:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.874194:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81383424 i_blocks 158976 00000020:00200000:1.0:1789760794.874196:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.874203:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.874226:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81383424, count: 4096 00000080:00200000:1.0:1789760794.874231:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81383424, bytes: 4096 00000080:00200000:1.0:1789760794.874237:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.874244:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.874246:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81383424, 81387520) 0 0 00000080:00200000:1.0:1789760794.874250:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.874322:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19869, 19869] 00020000:00200000:1.0:1789760794.874376:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81383424 chunk: [81383424, 81387520) 81387520, 4096 00020000:00200000:1.0:1789760794.874381:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.874396:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19869, 19869] 00000008:00010000:1.0:1789760794.874409:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.874414:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.874423:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81383424, 81387520) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.874435:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81383424=0x4d9d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.874475:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.874477:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81387520 00000020:00010000:1.0:1789760794.874482:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.874484:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.874486:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.874488:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.874607:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.874615:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.874622:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.874639:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760794.874719:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 81383424<>81387520 00000100:00100000:0.0:1789760794.874774:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503933440:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.892093:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987176 gen 2 00000008:00010000:0.0:1789760794.892129:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:0.0:1789760794.892136:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.892146:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.892209:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:0.0:1789760794.892232:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503933440:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.899804:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19869, 19869] 00000080:00200000:1.0:1789760794.899825:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.899831:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.899845:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81387520 i_blocks 158984 00000020:00200000:1.0:1789760794.899848:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.899861:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.899892:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81387520, count: 4096 00000080:00200000:1.0:1789760794.899899:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81387520, bytes: 4096 00000080:00200000:1.0:1789760794.899911:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.899918:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760794.899920:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81387520, 81391616) 0 0 00000080:00200000:1.0:1789760794.899923:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.899936:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19870, 19870] 00020000:00200000:1.0:1789760794.899944:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81387520 chunk: [81387520, 81391616) 81391616, 4096 00020000:00200000:1.0:1789760794.899948:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.899964:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19870, 19870] 00000008:00010000:1.0:1789760794.899978:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.899983:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760794.899992:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81387520, 81391616) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.900004:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81387520=0x4d9e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.900086:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.900089:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81391616 00000020:00010000:1.0:1789760794.900097:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.900100:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.900103:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.900106:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.900108:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.900116:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.900122:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.900144:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760794.900237:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 81387520<>81391616 00000100:00100000:0.0:1789760794.900617:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503933568:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.914595:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987177 gen 2 00000008:00010000:0.0:1789760794.914631:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d583c0 00000020:00200000:0.0:1789760794.914637:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.914646:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.914726:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d583c0 00000100:00100000:0.0:1789760794.914751:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503933568:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760794.915171:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19870, 19870] 00000080:00200000:1.0:1789760794.915183:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.915189:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.915317:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81391616 i_blocks 158992 00000020:00200000:0.0:1789760794.915321:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.915329:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.915356:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81391616, count: 4096 00000080:00200000:0.0:1789760794.915363:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81391616, bytes: 4096 00000080:00200000:0.0:1789760794.915371:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.915378:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.915380:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81391616, 81395712) 0 0 00000080:00200000:0.0:1789760794.915383:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.915395:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19871, 19871] 00020000:00200000:0.0:1789760794.915402:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81391616 chunk: [81391616, 81395712) 81395712, 4096 00020000:00200000:0.0:1789760794.915406:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.915422:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19871, 19871] 00000008:00010000:0.0:1789760794.915436:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.915441:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.915451:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81391616, 81395712) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.915461:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81391616=0x4d9f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.915587:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.915589:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81395712 00000020:00010000:0.0:1789760794.915598:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.915601:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.915604:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.915607:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d14183f8 (ffff9018c668f248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.915610:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.915685:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.915692:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.915706:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760794.915779:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b480 - object 0x280000401:2 offset 81391616<>81395712 00000100:00100000:1.0:1789760794.915835:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503933696:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760794.926889:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987178 gen 2 00000008:00010000:0.0:1789760794.926925:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:0.0:1789760794.926931:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.926940:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.926967:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:0.0:1789760794.926992:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503933696:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760794.927122:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19871, 19871] 00000080:00200000:3.0:1789760794.927133:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.927138:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.927150:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81395712 i_blocks 159000 00000020:00200000:3.0:1789760794.927153:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.927161:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.927207:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81395712, count: 4096 00000080:00200000:3.0:1789760794.927214:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81395712, bytes: 4096 00000080:00200000:3.0:1789760794.927221:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.927228:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760794.927230:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81395712, 81399808) 0 0 00000080:00200000:3.0:1789760794.927233:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.927245:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19872, 19872] 00020000:00200000:3.0:1789760794.927252:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81395712 chunk: [81395712, 81399808) 81399808, 4096 00020000:00200000:3.0:1789760794.927256:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.927269:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19872, 19872] 00000008:00010000:3.0:1789760794.927281:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.927286:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760794.927294:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81395712, 81399808) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.927305:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81395712=0x4da0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.927348:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.927555:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81399808 00000020:00010000:3.0:1789760794.927562:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.927576:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.927579:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.927582:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.927585:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.927592:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.927598:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.927612:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760794.927688:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070fb80 - object 0x280000401:2 offset 81395712<>81399808 00000100:00100000:3.0:1789760794.927727:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760794.927740:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503933824:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760794.941784:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987179 gen 2 00000008:00010000:3.0:1789760794.941818:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760794.941825:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.941833:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.941911:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760794.941936:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503933824:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.942017:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19872, 19872] 00000080:00200000:0.0:1789760794.942030:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.942060:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.942073:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81399808 i_blocks 159008 00000020:00200000:0.0:1789760794.942076:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.942084:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.942112:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81399808, count: 4096 00000080:00200000:0.0:1789760794.942118:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81399808, bytes: 4096 00000080:00200000:0.0:1789760794.942127:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.942134:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.942135:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81399808, 81403904) 0 0 00000080:00200000:0.0:1789760794.942139:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.942152:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19873, 19873] 00020000:00200000:0.0:1789760794.942158:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81399808 chunk: [81399808, 81403904) 81403904, 4096 00020000:00200000:0.0:1789760794.942162:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.942177:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19873, 19873] 00000008:00010000:0.0:1789760794.942345:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.942350:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.942359:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81399808, 81403904) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.942370:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81399808=0x4da1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.942911:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.942915:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81403904 00000020:00010000:0.0:1789760794.942923:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.942927:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.942930:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.942933:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.942936:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.942946:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.942953:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.942969:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760794.943084:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b480 - object 0x280000401:2 offset 81399808<>81403904 00000100:00100000:1.0:1789760794.943131:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503933952:192.168.203.158@tcp:4:dd.0 00010000:00010000:2.0:1789760794.943763:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760794.955958:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987180 gen 2 00000008:00010000:1.0:1789760794.955995:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760794.956001:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.956010:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.956028:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760794.956073:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503933952:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.956702:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19873, 19873] 00000080:00200000:0.0:1789760794.956717:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.956723:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.956740:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81403904 i_blocks 159016 00000020:00200000:0.0:1789760794.956744:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.956753:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.956789:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81403904, count: 4096 00000080:00200000:0.0:1789760794.956796:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81403904, bytes: 4096 00000080:00200000:0.0:1789760794.956809:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.956817:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.956820:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81403904, 81408000) 0 0 00000080:00200000:0.0:1789760794.956823:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.956838:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19874, 19874] 00020000:00200000:0.0:1789760794.956845:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81403904 chunk: [81403904, 81408000) 81408000, 4096 00020000:00200000:0.0:1789760794.956849:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.956863:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19874, 19874] 00000008:00010000:0.0:1789760794.956876:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.956880:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.956889:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81403904, 81408000) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.956901:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81403904=0x4da2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.956947:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.956951:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81408000 00000020:00010000:0.0:1789760794.956957:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.956961:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.956964:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.956966:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.956970:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.956977:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.956983:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.956998:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760794.957099:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a680 - object 0x280000401:2 offset 81403904<>81408000 00000100:00100000:1.0:1789760794.957197:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503934080:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.965995:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987181 gen 2 00000008:00010000:1.0:1789760794.966031:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760794.966060:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.966069:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.966084:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760794.966103:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503934080:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.966135:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19874, 19874] 00000080:00200000:0.0:1789760794.966148:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.966152:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.966165:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81408000 i_blocks 159024 00000020:00200000:0.0:1789760794.966168:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.966176:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.966200:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81408000, count: 4096 00000080:00200000:0.0:1789760794.966205:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81408000, bytes: 4096 00000080:00200000:0.0:1789760794.966212:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.966219:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.966221:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81408000, 81412096) 0 0 00000080:00200000:0.0:1789760794.966225:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.966237:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19875, 19875] 00020000:00200000:0.0:1789760794.966244:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81408000 chunk: [81408000, 81412096) 81412096, 4096 00020000:00200000:0.0:1789760794.966247:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.966261:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19875, 19875] 00000008:00010000:0.0:1789760794.966273:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.966279:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.966287:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81408000, 81412096) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.966305:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81408000=0x4da3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.966347:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.966350:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81412096 00000020:00010000:0.0:1789760794.966356:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.966360:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.966362:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.966365:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.966367:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.966374:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.966380:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.966393:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760794.966467:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98000 - object 0x280000401:2 offset 81408000<>81412096 00000100:00100000:1.0:1789760794.966580:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503934208:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.976522:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987182 gen 2 00000008:00010000:1.0:1789760794.976552:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760794.976557:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.976574:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.976630:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760794.976655:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503934208:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760794.976718:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19875, 19875] 00000080:00200000:0.0:1789760794.976728:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.976732:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.976743:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81412096 i_blocks 159032 00000020:00200000:0.0:1789760794.976745:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.976751:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.976773:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81412096, count: 4096 00000080:00200000:0.0:1789760794.976777:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81412096, bytes: 4096 00000080:00200000:0.0:1789760794.976782:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.976788:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760794.976789:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81412096, 81416192) 0 0 00000080:00200000:0.0:1789760794.976792:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.976802:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19876, 19876] 00020000:00200000:0.0:1789760794.976808:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81412096 chunk: [81412096, 81416192) 81416192, 4096 00020000:00200000:0.0:1789760794.976810:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.976823:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19876, 19876] 00000008:00010000:0.0:1789760794.976833:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.976837:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760794.976844:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81412096, 81416192) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.976852:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81412096=0x4da4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.976901:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.976905:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81416192 00000020:00010000:0.0:1789760794.976923:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.976927:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.976930:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.976933:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.976936:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.976942:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.976948:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.976962:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760794.977196:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98a80 - object 0x280000401:2 offset 81412096<>81416192 00000100:00100000:1.0:1789760794.977253:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503934336:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760794.990701:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987183 gen 2 00000008:00010000:1.0:1789760794.990736:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760794.990743:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.990752:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.990820:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760794.990846:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503934336:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760794.991187:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19876, 19876] 00000080:00200000:3.0:1789760794.991200:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.991205:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.991217:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81416192 i_blocks 159040 00000020:00200000:3.0:1789760794.991221:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.991229:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.991365:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81416192, count: 4096 00000080:00200000:3.0:1789760794.991372:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81416192, bytes: 4096 00000080:00200000:3.0:1789760794.991381:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.991388:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760794.991390:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81416192, 81420288) 0 0 00000080:00200000:3.0:1789760794.991394:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.991406:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19877, 19877] 00020000:00200000:3.0:1789760794.991414:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81416192 chunk: [81416192, 81420288) 81420288, 4096 00020000:00200000:3.0:1789760794.991418:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.991433:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19877, 19877] 00000008:00010000:3.0:1789760794.991447:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.991451:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760794.991461:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81416192, 81420288) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.991472:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81416192=0x4da5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.991516:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.991519:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81420288 00000020:00010000:3.0:1789760794.991525:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.991528:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.991531:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.991533:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.991536:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.991542:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.991547:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.991561:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760794.991714:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070fb80 - object 0x280000401:2 offset 81416192<>81420288 00000100:00100000:2.0:1789760794.991783:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503934464:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.013917:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987184 gen 2 00000008:00010000:3.0:1789760795.013952:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760795.013958:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.013967:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.014062:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760795.014081:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503934464:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760795.015454:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19877, 19877] 00000080:00200000:1.0:1789760795.015471:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.015476:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.015489:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81420288 i_blocks 159048 00000020:00200000:1.0:1789760795.015493:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.015504:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.015529:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81420288, count: 4096 00000080:00200000:1.0:1789760795.015535:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81420288, bytes: 4096 00000080:00200000:1.0:1789760795.015545:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.015553:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760795.015555:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81420288, 81424384) 0 0 00000080:00200000:1.0:1789760795.015559:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.015582:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19878, 19878] 00020000:00200000:1.0:1789760795.015588:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81420288 chunk: [81420288, 81424384) 81424384, 4096 00020000:00200000:1.0:1789760795.015592:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.015617:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19878, 19878] 00000008:00010000:1.0:1789760795.015631:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.015635:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760795.015643:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81420288, 81424384) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.015671:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81420288=0x4da6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.015720:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.015723:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81424384 00000020:00010000:1.0:1789760795.015729:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.015732:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.015735:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.015737:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.015740:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.015745:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.015750:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.015769:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760795.015848:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81420288<>81424384 00000100:00100000:1.0:1789760795.015889:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760795.015907:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503934592:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760795.027372:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987185 gen 2 00000008:00010000:1.0:1789760795.027407:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58540 00000020:00200000:1.0:1789760795.027413:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.027421:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.027444:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58540 00000100:00100000:1.0:1789760795.027467:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503934592:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760795.027560:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19878, 19878] 00000080:00200000:3.0:1789760795.027581:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.027587:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.027599:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81424384 i_blocks 159056 00000020:00200000:3.0:1789760795.027610:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.027619:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.027642:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81424384, count: 4096 00000080:00200000:3.0:1789760795.027648:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81424384, bytes: 4096 00000080:00200000:3.0:1789760795.027839:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.027846:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760795.027848:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81424384, 81428480) 0 0 00000080:00200000:3.0:1789760795.027852:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.027864:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19879, 19879] 00020000:00200000:3.0:1789760795.027871:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81424384 chunk: [81424384, 81428480) 81428480, 4096 00020000:00200000:3.0:1789760795.027875:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.027891:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19879, 19879] 00000008:00010000:3.0:1789760795.027905:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.027910:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760795.027920:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81424384, 81428480) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.027931:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81424384=0x4da7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.027977:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.027980:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81428480 00000020:00010000:3.0:1789760795.027986:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.027990:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.027992:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.027996:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.027998:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.028004:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.028010:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.028023:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760795.028307:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070fb80 - object 0x280000401:2 offset 81424384<>81428480 00000100:00100000:2.0:1789760795.028492:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760795.029084:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503934720:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.037314:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987186 gen 2 00000008:00010000:3.0:1789760795.037348:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760795.037354:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.037363:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.037381:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760795.037405:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503934720:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.037593:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19879, 19879] 00000080:00200000:2.0:1789760795.037619:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.037625:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.037638:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81428480 i_blocks 159064 00000020:00200000:2.0:1789760795.037641:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.037649:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.037675:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81428480, count: 4096 00000080:00200000:2.0:1789760795.037680:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81428480, bytes: 4096 00000080:00200000:2.0:1789760795.037688:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.037694:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.037697:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81428480, 81432576) 0 0 00000080:00200000:2.0:1789760795.037701:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.037714:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19880, 19880] 00020000:00200000:2.0:1789760795.037720:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81428480 chunk: [81428480, 81432576) 81432576, 4096 00020000:00200000:2.0:1789760795.037724:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.037739:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19880, 19880] 00000008:00010000:2.0:1789760795.037752:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.037757:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.037766:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81428480, 81432576) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.037776:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81428480=0x4da8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.037821:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.037824:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81432576 00000020:00010000:2.0:1789760795.037831:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.037834:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.037836:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.037839:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.037841:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.037847:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.037852:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.037865:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760795.037939:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25f80 - object 0x280000401:2 offset 81428480<>81432576 00000100:00100000:3.0:1789760795.038091:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760795.038106:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503934848:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.049555:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987187 gen 2 00000008:00010000:3.0:1789760795.049618:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760795.049624:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.049633:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.049651:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760795.049673:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503934848:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.049709:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19880, 19880] 00000080:00200000:2.0:1789760795.049723:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.049728:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.049740:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81432576 i_blocks 159072 00000020:00200000:2.0:1789760795.049744:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.049751:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.049776:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81432576, count: 4096 00000080:00200000:2.0:1789760795.049782:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81432576, bytes: 4096 00000080:00200000:2.0:1789760795.049789:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.049796:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.049798:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81432576, 81436672) 0 0 00000080:00200000:2.0:1789760795.049801:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.049813:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19881, 19881] 00020000:00200000:2.0:1789760795.049821:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81432576 chunk: [81432576, 81436672) 81436672, 4096 00020000:00200000:2.0:1789760795.049824:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.049839:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19881, 19881] 00000008:00010000:2.0:1789760795.049851:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.049856:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.049864:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81432576, 81436672) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.049873:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81432576=0x4da9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.049912:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.049925:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81436672 00000020:00010000:2.0:1789760795.049932:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.049935:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.049938:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.049941:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.049944:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.049950:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.049955:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.049968:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c240 00000008:00100000:2.0:1789760795.050214:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26300 - object 0x280000401:2 offset 81432576<>81436672 00000100:00100000:2.0:1789760795.050253:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503934976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.065864:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987188 gen 2 00000008:00010000:3.0:1789760795.065900:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760795.065906:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.065915:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.065935:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760795.065956:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503934976:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760795.066573:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19881, 19881] 00000080:00200000:1.0:1789760795.066588:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.066594:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.066607:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81436672 i_blocks 159080 00000020:00200000:1.0:1789760795.066610:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.066619:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.066647:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81436672, count: 4096 00000080:00200000:1.0:1789760795.066653:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81436672, bytes: 4096 00000080:00200000:1.0:1789760795.066662:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.066668:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760795.066670:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81436672, 81440768) 0 0 00000080:00200000:1.0:1789760795.066673:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.066685:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19882, 19882] 00020000:00200000:1.0:1789760795.066692:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81436672 chunk: [81436672, 81440768) 81440768, 4096 00020000:00200000:1.0:1789760795.066696:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.066711:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19882, 19882] 00000008:00010000:1.0:1789760795.066725:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.066730:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760795.066740:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81436672, 81440768) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.066751:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81436672=0x4daa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.067288:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.067293:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81440768 00000020:00010000:1.0:1789760795.067302:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.067306:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.067309:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.067312:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.067315:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.067322:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.067329:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.067347:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760795.067435:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81436672<>81440768 00000100:00100000:0.0:1789760795.067890:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503935104:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760795.102450:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987189 gen 2 00000008:00010000:0.0:1789760795.102653:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58540 00000020:00200000:0.0:1789760795.102660:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.102669:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.102694:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58540 00000100:00100000:0.0:1789760795.102712:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503935104:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.102888:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19882, 19882] 00000080:00200000:2.0:1789760795.102900:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.102905:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.102924:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81440768 i_blocks 159088 00000020:00200000:2.0:1789760795.102928:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.102936:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.102958:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81440768, count: 4096 00000080:00200000:2.0:1789760795.102964:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81440768, bytes: 4096 00000080:00200000:2.0:1789760795.102974:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.102981:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.102983:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81440768, 81444864) 0 0 00000080:00200000:2.0:1789760795.102987:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.102999:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19883, 19883] 00020000:00200000:2.0:1789760795.103006:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81440768 chunk: [81440768, 81444864) 81444864, 4096 00020000:00200000:2.0:1789760795.103010:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.103023:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19883, 19883] 00000008:00010000:2.0:1789760795.103057:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.103062:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.103071:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81440768, 81444864) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.103092:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81440768=0x4dab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.103133:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.103135:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81444864 00000020:00010000:2.0:1789760795.103143:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.103146:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.103149:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.103152:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.103155:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.103162:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.103168:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.103179:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c900 00000008:00100000:2.0:1789760795.103254:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24a80 - object 0x280000401:2 offset 81440768<>81444864 00000100:00100000:2.0:1789760795.103292:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503935232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.113734:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987190 gen 2 00000008:00010000:3.0:1789760795.113779:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760795.113785:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.113795:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.113813:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760795.113838:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503935232:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.114184:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19883, 19883] 00000080:00200000:2.0:1789760795.114199:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.114204:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.114216:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81444864 i_blocks 159096 00000020:00200000:2.0:1789760795.114219:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.114227:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.114253:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81444864, count: 4096 00000080:00200000:2.0:1789760795.114260:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81444864, bytes: 4096 00000080:00200000:2.0:1789760795.114268:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.114277:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.114278:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81444864, 81448960) 0 0 00000080:00200000:2.0:1789760795.114282:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.114294:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19884, 19884] 00020000:00200000:2.0:1789760795.114301:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81444864 chunk: [81444864, 81448960) 81448960, 4096 00020000:00200000:2.0:1789760795.114305:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.114320:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19884, 19884] 00000008:00010000:2.0:1789760795.114333:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.114338:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.114347:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81444864, 81448960) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.114358:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81444864=0x4dac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.114400:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.114403:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81448960 00000020:00010000:2.0:1789760795.114410:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.114414:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.114417:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.114419:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.114422:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.114428:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.114433:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.114446:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000008:00100000:2.0:1789760795.114640:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27100 - object 0x280000401:2 offset 81444864<>81448960 00000100:00100000:3.0:1789760795.114733:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503935360:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.132815:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987191 gen 2 00000008:00010000:3.0:1789760795.132853:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760795.132859:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.132867:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.132886:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760795.132920:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503935360:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.132930:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19884, 19884] 00000080:00200000:2.0:1789760795.132943:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.132949:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.132962:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81448960 i_blocks 159104 00000020:00200000:2.0:1789760795.132965:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.132973:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.132996:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81448960, count: 4096 00000080:00200000:2.0:1789760795.133002:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81448960, bytes: 4096 00000080:00200000:2.0:1789760795.133008:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.133015:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.133017:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81448960, 81453056) 0 0 00000080:00200000:2.0:1789760795.133020:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.133032:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19885, 19885] 00020000:00200000:2.0:1789760795.133068:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81448960 chunk: [81448960, 81453056) 81453056, 4096 00020000:00200000:2.0:1789760795.133072:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.133605:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19885, 19885] 00000008:00010000:2.0:1789760795.133622:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.133627:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.133637:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81448960, 81453056) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.133648:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81448960=0x4dad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.133694:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.133698:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81453056 00000020:00010000:2.0:1789760795.133705:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.133708:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.133711:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.133714:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.133717:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.133725:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.133732:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.133751:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cb40 00000008:00100000:2.0:1789760795.133832:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27100 - object 0x280000401:2 offset 81448960<>81453056 00000100:00100000:3.0:1789760795.134189:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760795.134207:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503935488:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.147372:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987192 gen 2 00000008:00010000:3.0:1789760795.147413:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760795.147420:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.147430:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.147523:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760795.147547:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503935488:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.150318:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19885, 19885] 00000080:00200000:2.0:1789760795.150337:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.150343:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.150357:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81453056 i_blocks 159112 00000020:00200000:2.0:1789760795.150361:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.150373:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.150398:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81453056, count: 4096 00000080:00200000:2.0:1789760795.150405:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81453056, bytes: 4096 00000080:00200000:2.0:1789760795.150415:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.150423:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.150425:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81453056, 81457152) 0 0 00000080:00200000:2.0:1789760795.150428:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.150440:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19886, 19886] 00020000:00200000:2.0:1789760795.150446:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81453056 chunk: [81453056, 81457152) 81457152, 4096 00020000:00200000:2.0:1789760795.150450:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.150465:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19886, 19886] 00000008:00010000:2.0:1789760795.150479:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.150484:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.150493:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81453056, 81457152) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.151406:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81453056=0x4dae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.151468:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.151471:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81457152 00000020:00010000:2.0:1789760795.151478:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.151481:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.151484:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.151487:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.151489:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.151498:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.151586:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.151605:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c600 00000008:00100000:2.0:1789760795.151683:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24a80 - object 0x280000401:2 offset 81453056<>81457152 00000100:00100000:3.0:1789760795.151753:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503935616:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.168423:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987193 gen 2 00000008:00010000:3.0:1789760795.168460:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760795.168466:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.168475:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.168492:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760795.168513:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503935616:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.169188:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19886, 19886] 00000080:00200000:2.0:1789760795.169204:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.169210:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.169222:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81457152 i_blocks 159120 00000020:00200000:2.0:1789760795.169225:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.169234:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.169258:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81457152, count: 4096 00000080:00200000:2.0:1789760795.169263:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81457152, bytes: 4096 00000080:00200000:2.0:1789760795.169272:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.169279:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.169281:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81457152, 81461248) 0 0 00000080:00200000:2.0:1789760795.169284:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.169296:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19887, 19887] 00020000:00200000:2.0:1789760795.169302:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81457152 chunk: [81457152, 81461248) 81461248, 4096 00020000:00200000:2.0:1789760795.169305:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.169319:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19887, 19887] 00000008:00010000:2.0:1789760795.169333:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.169337:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.169345:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81457152, 81461248) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.169357:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81457152=0x4daf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.169402:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.169405:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81461248 00000020:00010000:2.0:1789760795.169411:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.169414:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.169417:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.169421:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.169423:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.169430:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.169436:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.169451:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c540 00000008:00100000:2.0:1789760795.169527:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26300 - object 0x280000401:2 offset 81457152<>81461248 00000100:00100000:3.0:1789760795.170256:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503935744:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.189532:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987194 gen 2 00000008:00010000:3.0:1789760795.189582:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760795.189590:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.189599:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.189622:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760795.189647:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503935744:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.189776:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19887, 19887] 00000080:00200000:2.0:1789760795.189789:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.189793:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.189806:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81461248 i_blocks 159128 00000020:00200000:2.0:1789760795.189809:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.189818:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.189843:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81461248, count: 4096 00000080:00200000:2.0:1789760795.189848:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81461248, bytes: 4096 00000080:00200000:2.0:1789760795.189855:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.189861:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.189863:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81461248, 81465344) 0 0 00000080:00200000:2.0:1789760795.189866:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.189878:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19888, 19888] 00020000:00200000:2.0:1789760795.189885:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81461248 chunk: [81461248, 81465344) 81465344, 4096 00020000:00200000:2.0:1789760795.189888:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.189901:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19888, 19888] 00000008:00010000:2.0:1789760795.189915:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.189920:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.189929:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81461248, 81465344) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.189940:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81461248=0x4db0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.189984:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.189986:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81465344 00000020:00010000:2.0:1789760795.189993:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.189996:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.190000:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.190002:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.190005:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.190012:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.190018:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.190066:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000008:00100000:2.0:1789760795.190324:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25f80 - object 0x280000401:2 offset 81461248<>81465344 00000100:00100000:3.0:1789760795.190374:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503935872:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.224409:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987195 gen 2 00000008:00010000:2.0:1789760795.224713:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:2.0:1789760795.224720:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.224728:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.224746:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:2.0:1789760795.224767:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503935872:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760795.225192:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19888, 19888] 00000080:00200000:3.0:1789760795.225206:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.225211:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.225224:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81465344 i_blocks 159136 00000020:00200000:3.0:1789760795.225227:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.225236:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.225262:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81465344, count: 4096 00000080:00200000:3.0:1789760795.225268:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81465344, bytes: 4096 00000080:00200000:3.0:1789760795.225279:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.225358:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760795.225361:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81465344, 81469440) 0 0 00000080:00200000:3.0:1789760795.225367:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.225380:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19889, 19889] 00020000:00200000:3.0:1789760795.225388:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81465344 chunk: [81465344, 81469440) 81469440, 4096 00020000:00200000:3.0:1789760795.225391:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.225406:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19889, 19889] 00000008:00010000:3.0:1789760795.225419:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.225424:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760795.225431:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81465344, 81469440) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.225441:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81465344=0x4db1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.225486:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.225488:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81469440 00000020:00010000:3.0:1789760795.225494:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.225497:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.225500:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.225503:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.225506:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.225512:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.225518:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.225534:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760795.225988:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070fb80 - object 0x280000401:2 offset 81465344<>81469440 00000100:00100000:2.0:1789760795.226410:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503936000:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.240792:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987196 gen 2 00000008:00010000:3.0:1789760795.240833:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760795.240840:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.240848:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.240869:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760795.240893:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503936000:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760795.241195:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19889, 19889] 00000080:00200000:1.0:1789760795.241210:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.241215:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.241228:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81469440 i_blocks 159144 00000020:00200000:1.0:1789760795.241231:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.241239:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.241265:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81469440, count: 4096 00000080:00200000:1.0:1789760795.241272:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81469440, bytes: 4096 00000080:00200000:1.0:1789760795.241280:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.241298:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760795.241301:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81469440, 81473536) 0 0 00000080:00200000:1.0:1789760795.241305:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.241318:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19890, 19890] 00020000:00200000:1.0:1789760795.241324:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81469440 chunk: [81469440, 81473536) 81473536, 4096 00020000:00200000:1.0:1789760795.241328:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.241342:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19890, 19890] 00000008:00010000:1.0:1789760795.241357:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.241361:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760795.241370:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81469440, 81473536) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.241380:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81469440=0x4db2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.241426:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.241429:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81473536 00000020:00010000:1.0:1789760795.241435:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.241438:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.241441:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.241443:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.241446:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.241452:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.241458:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.241473:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760795.241576:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81469440<>81473536 00000100:00100000:0.0:1789760795.241647:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503936128:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760795.264375:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987197 gen 2 00000008:00010000:0.0:1789760795.264409:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58a80 00000020:00200000:0.0:1789760795.264416:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.264425:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.264447:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58a80 00000100:00100000:0.0:1789760795.264471:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503936128:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760795.266461:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19890, 19890] 00000080:00200000:1.0:1789760795.266482:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.266487:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.266511:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81473536 i_blocks 159152 00000020:00200000:1.0:1789760795.266516:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.266528:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.266556:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81473536, count: 4096 00000080:00200000:1.0:1789760795.266572:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81473536, bytes: 4096 00000080:00200000:1.0:1789760795.266583:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.266592:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760795.266594:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81473536, 81477632) 0 0 00000080:00200000:1.0:1789760795.266598:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.266612:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19891, 19891] 00020000:00200000:1.0:1789760795.266620:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81473536 chunk: [81473536, 81477632) 81477632, 4096 00020000:00200000:1.0:1789760795.266623:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.266638:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19891, 19891] 00000008:00010000:1.0:1789760795.266651:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.266656:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760795.266665:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81473536, 81477632) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.266677:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81473536=0x4db3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.266728:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.266731:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81477632 00000020:00010000:1.0:1789760795.266737:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.266740:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.266743:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.266746:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.266749:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.266755:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.266766:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.266784:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760795.266867:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 81473536<>81477632 00000100:00100000:0.0:1789760795.266940:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503936256:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760795.282579:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987198 gen 2 00000008:00010000:0.0:1789760795.282627:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d586c0 00000020:00200000:0.0:1789760795.282635:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.282643:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.282661:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d586c0 00000100:00100000:0.0:1789760795.282685:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503936256:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.282819:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19891, 19891] 00000080:00200000:2.0:1789760795.282832:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.282836:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.282848:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81477632 i_blocks 159160 00000020:00200000:2.0:1789760795.282851:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.282860:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.282885:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81477632, count: 4096 00000080:00200000:2.0:1789760795.282891:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81477632, bytes: 4096 00000080:00200000:2.0:1789760795.282898:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.282905:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.282907:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81477632, 81481728) 0 0 00000080:00200000:2.0:1789760795.282911:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.282922:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19892, 19892] 00020000:00200000:2.0:1789760795.282928:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81477632 chunk: [81477632, 81481728) 81481728, 4096 00020000:00200000:2.0:1789760795.282932:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.282946:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19892, 19892] 00000008:00010000:2.0:1789760795.282960:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.282964:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.282972:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81477632, 81481728) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.282984:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81477632=0x4db4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.283029:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.283032:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81481728 00000020:00010000:2.0:1789760795.283072:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.283075:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.283078:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.283081:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.283084:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.283091:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.283096:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.283109:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000008:00100000:2.0:1789760795.283183:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25f80 - object 0x280000401:2 offset 81477632<>81481728 00000100:00100000:3.0:1789760795.283275:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760795.283297:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503936384:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.294367:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987199 gen 2 00000008:00010000:3.0:1789760795.294400:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760795.294407:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.294416:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.294438:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760795.294460:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503936384:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.294554:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19892, 19892] 00000080:00200000:2.0:1789760795.294578:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.294583:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.294596:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81481728 i_blocks 159168 00000020:00200000:2.0:1789760795.294599:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.294669:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.294700:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81481728, count: 4096 00000080:00200000:2.0:1789760795.294706:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81481728, bytes: 4096 00000080:00200000:2.0:1789760795.294714:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.294721:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.294723:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81481728, 81485824) 0 0 00000080:00200000:2.0:1789760795.294727:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.294739:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19893, 19893] 00020000:00200000:2.0:1789760795.294745:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81481728 chunk: [81481728, 81485824) 81485824, 4096 00020000:00200000:2.0:1789760795.294749:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.294763:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19893, 19893] 00000008:00010000:2.0:1789760795.294776:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.294781:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.294789:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81481728, 81485824) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.294799:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81481728=0x4db5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.294843:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.294846:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81485824 00000020:00010000:2.0:1789760795.294852:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.294855:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.294858:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.294861:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.294863:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.294871:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.294877:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.294891:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c840 00000008:00100000:2.0:1789760795.294969:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26a00 - object 0x280000401:2 offset 81481728<>81485824 00000100:00100000:3.0:1789760795.295106:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503936512:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.311297:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987200 gen 2 00000008:00010000:3.0:1789760795.311335:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760795.311341:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.311350:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.311412:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760795.311436:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503936512:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.312077:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19893, 19893] 00000080:00200000:2.0:1789760795.312090:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.312095:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.312107:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81485824 i_blocks 159176 00000020:00200000:2.0:1789760795.312111:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.312118:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.312143:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81485824, count: 4096 00000080:00200000:2.0:1789760795.312148:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81485824, bytes: 4096 00000080:00200000:2.0:1789760795.312156:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.312163:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.312166:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81485824, 81489920) 0 0 00000080:00200000:2.0:1789760795.312170:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.312181:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19894, 19894] 00020000:00200000:2.0:1789760795.312187:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81485824 chunk: [81485824, 81489920) 81489920, 4096 00020000:00200000:2.0:1789760795.312191:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.312204:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19894, 19894] 00000008:00010000:2.0:1789760795.312216:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.312221:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.312230:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81485824, 81489920) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.312240:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81485824=0x4db6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.312286:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.312289:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81489920 00000020:00010000:2.0:1789760795.312296:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.312299:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.312302:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.312305:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.312307:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.312314:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.312320:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.312336:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cd80 00000008:00100000:2.0:1789760795.312406:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27800 - object 0x280000401:2 offset 81485824<>81489920 00000100:00100000:3.0:1789760795.312612:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503936640:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.328536:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987201 gen 2 00000008:00010000:3.0:1789760795.328580:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760795.328587:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.328595:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.328799:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760795.328824:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503936640:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.332294:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19894, 19894] 00000080:00200000:2.0:1789760795.332313:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.332318:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.332332:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81489920 i_blocks 159184 00000020:00200000:2.0:1789760795.332336:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.332347:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.332372:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81489920, count: 4096 00000080:00200000:2.0:1789760795.332379:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81489920, bytes: 4096 00000080:00200000:2.0:1789760795.332389:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.332396:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.332398:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81489920, 81494016) 0 0 00000080:00200000:2.0:1789760795.332402:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.332413:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19895, 19895] 00020000:00200000:2.0:1789760795.332419:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81489920 chunk: [81489920, 81494016) 81494016, 4096 00020000:00200000:2.0:1789760795.332423:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.332437:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19895, 19895] 00000008:00010000:2.0:1789760795.332448:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.332452:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.332460:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81489920, 81494016) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.332480:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81489920=0x4db7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.332539:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.332542:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81494016 00000020:00010000:2.0:1789760795.332549:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.332552:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.332555:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.332557:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.332570:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.332578:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.332584:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.332604:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c180 00000008:00100000:2.0:1789760795.332704:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25180 - object 0x280000401:2 offset 81489920<>81494016 00000100:00100000:3.0:1789760795.332789:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503936768:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.341863:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987202 gen 2 00000008:00010000:3.0:1789760795.341899:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760795.341907:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.341915:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.341972:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760795.341996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503936768:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.342207:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19895, 19895] 00000080:00200000:2.0:1789760795.342220:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.342225:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.342236:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81494016 i_blocks 159192 00000020:00200000:2.0:1789760795.342240:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.342248:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.342271:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81494016, count: 4096 00000080:00200000:2.0:1789760795.342277:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81494016, bytes: 4096 00000080:00200000:2.0:1789760795.342286:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.342292:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.342295:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81494016, 81498112) 0 0 00000080:00200000:2.0:1789760795.342299:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.342311:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19896, 19896] 00020000:00200000:2.0:1789760795.342317:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81494016 chunk: [81494016, 81498112) 81498112, 4096 00020000:00200000:2.0:1789760795.342320:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.342334:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19896, 19896] 00000008:00010000:2.0:1789760795.342346:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.342350:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.342358:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81494016, 81498112) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.342368:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81494016=0x4db8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.342409:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.342412:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81498112 00000020:00010000:2.0:1789760795.342419:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.342422:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.342425:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.342428:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.342430:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.342437:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.342442:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.342454:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c780 00000008:00100000:2.0:1789760795.342526:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25c00 - object 0x280000401:2 offset 81494016<>81498112 00000100:00100000:3.0:1789760795.342732:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503936896:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.349847:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987203 gen 2 00000008:00010000:3.0:1789760795.349881:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760795.349887:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.349894:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.349914:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760795.349937:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503936896:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.350022:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19896, 19896] 00000080:00200000:2.0:1789760795.350117:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.350124:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.350138:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81498112 i_blocks 159200 00000020:00200000:2.0:1789760795.350142:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.350150:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.350181:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81498112, count: 4096 00000080:00200000:2.0:1789760795.350188:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81498112, bytes: 4096 00000080:00200000:2.0:1789760795.350195:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.350202:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.350204:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81498112, 81502208) 0 0 00000080:00200000:2.0:1789760795.350208:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.350220:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19897, 19897] 00020000:00200000:2.0:1789760795.350227:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81498112 chunk: [81498112, 81502208) 81502208, 4096 00020000:00200000:2.0:1789760795.350230:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.350244:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19897, 19897] 00000008:00010000:2.0:1789760795.350258:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.350263:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.350272:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81498112, 81502208) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.350282:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81498112=0x4db9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.350324:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.350327:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81502208 00000020:00010000:2.0:1789760795.350333:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.350336:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.350338:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.350341:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.350344:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.350350:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.350356:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.350369:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c780 00000008:00100000:2.0:1789760795.350449:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24380 - object 0x280000401:2 offset 81498112<>81502208 00000100:00100000:2.0:1789760795.350502:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503937024:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.365007:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987204 gen 2 00000008:00010000:3.0:1789760795.365081:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760795.365088:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.365098:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.365126:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760795.365157:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503937024:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.365464:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19897, 19897] 00000080:00200000:2.0:1789760795.365476:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.365481:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.365502:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81502208 i_blocks 159208 00000020:00200000:2.0:1789760795.365505:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.365513:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.365537:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81502208, count: 4096 00000080:00200000:2.0:1789760795.365543:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81502208, bytes: 4096 00000080:00200000:2.0:1789760795.365552:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.365558:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.365570:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81502208, 81506304) 0 0 00000080:00200000:2.0:1789760795.365574:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.365586:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19898, 19898] 00020000:00200000:2.0:1789760795.365593:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81502208 chunk: [81502208, 81506304) 81506304, 4096 00020000:00200000:2.0:1789760795.365607:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.365623:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19898, 19898] 00000008:00010000:2.0:1789760795.365637:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.365642:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.365652:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81502208, 81506304) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.365662:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81502208=0x4dba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.365716:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.365719:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81506304 00000020:00010000:2.0:1789760795.365726:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.365729:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.365732:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.365735:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.365737:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.365743:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.365749:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.365763:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c180 00000008:00100000:2.0:1789760795.365841:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24700 - object 0x280000401:2 offset 81502208<>81506304 00000100:00100000:3.0:1789760795.366791:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760795.366814:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503937152:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.381282:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987205 gen 2 00000008:00010000:3.0:1789760795.381318:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760795.381324:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.381332:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.381388:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760795.381411:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503937152:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.382970:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19898, 19898] 00000080:00200000:2.0:1789760795.382988:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.382994:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.383008:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81506304 i_blocks 159216 00000020:00200000:2.0:1789760795.383012:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.383024:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.383064:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81506304, count: 4096 00000080:00200000:2.0:1789760795.383071:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81506304, bytes: 4096 00000080:00200000:2.0:1789760795.383082:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.383089:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.383091:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81506304, 81510400) 0 0 00000080:00200000:2.0:1789760795.383095:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.383107:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19899, 19899] 00020000:00200000:2.0:1789760795.383114:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81506304 chunk: [81506304, 81510400) 81510400, 4096 00020000:00200000:2.0:1789760795.383118:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.383133:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19899, 19899] 00000008:00010000:2.0:1789760795.383146:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.383151:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.383160:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81506304, 81510400) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.383171:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81506304=0x4dbb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.383223:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.383226:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81510400 00000020:00010000:2.0:1789760795.383232:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.383235:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.383493:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.383496:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.383499:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.383507:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.383513:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.383532:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cd80 00000008:00100000:2.0:1789760795.383686:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 81506304<>81510400 00000100:00100000:3.0:1789760795.383761:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503937280:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.398346:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987206 gen 2 00000008:00010000:3.0:1789760795.398384:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760795.398391:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.398399:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.398452:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760795.398475:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503937280:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.398794:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19899, 19899] 00000080:00200000:2.0:1789760795.398807:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.398812:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.398824:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81510400 i_blocks 159224 00000020:00200000:2.0:1789760795.398827:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.398835:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.398860:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81510400, count: 4096 00000080:00200000:2.0:1789760795.398866:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81510400, bytes: 4096 00000080:00200000:2.0:1789760795.398930:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.398939:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.398940:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81510400, 81514496) 0 0 00000080:00200000:2.0:1789760795.398944:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.398957:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19900, 19900] 00020000:00200000:2.0:1789760795.398964:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81510400 chunk: [81510400, 81514496) 81514496, 4096 00020000:00200000:2.0:1789760795.398968:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.398983:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19900, 19900] 00000008:00010000:2.0:1789760795.398997:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.399002:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.399011:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81510400, 81514496) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.399022:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81510400=0x4dbc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.399089:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.399092:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81514496 00000020:00010000:2.0:1789760795.399099:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.399102:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.399105:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.399108:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.399111:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.399118:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.399124:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.399211:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c840 00000008:00100000:2.0:1789760795.399361:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24e00 - object 0x280000401:2 offset 81510400<>81514496 00000100:00100000:3.0:1789760795.399490:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503937408:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.408851:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987207 gen 2 00000008:00010000:3.0:1789760795.408889:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760795.408896:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.408905:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.408926:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760795.408950:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503937408:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.409715:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19900, 19900] 00000080:00200000:2.0:1789760795.409730:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.409735:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.409747:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81514496 i_blocks 159232 00000020:00200000:2.0:1789760795.409751:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.409761:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.409786:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81514496, count: 4096 00000080:00200000:2.0:1789760795.409792:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81514496, bytes: 4096 00000080:00200000:2.0:1789760795.409800:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.409807:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.409809:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81514496, 81518592) 0 0 00000080:00200000:2.0:1789760795.409812:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.409824:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19901, 19901] 00020000:00200000:2.0:1789760795.409831:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81514496 chunk: [81514496, 81518592) 81518592, 4096 00020000:00200000:2.0:1789760795.409835:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.409850:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19901, 19901] 00000008:00010000:2.0:1789760795.409863:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.409868:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.409877:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81514496, 81518592) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.409887:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81514496=0x4dbd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.409934:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.409937:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81518592 00000020:00010000:2.0:1789760795.409944:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.409947:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.409950:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.409954:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.409956:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.409964:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.409969:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.409984:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000008:00100000:2.0:1789760795.410106:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27480 - object 0x280000401:2 offset 81514496<>81518592 00000100:00100000:3.0:1789760795.410488:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760795.410588:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503937536:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.423909:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987208 gen 2 00000008:00010000:3.0:1789760795.423948:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760795.423955:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.423963:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.423983:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760795.424008:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503937536:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.424239:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19901, 19901] 00000080:00200000:2.0:1789760795.424252:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.424258:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.424270:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81518592 i_blocks 159240 00000020:00200000:2.0:1789760795.424273:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.424280:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.424317:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81518592, count: 4096 00000080:00200000:2.0:1789760795.424324:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81518592, bytes: 4096 00000080:00200000:2.0:1789760795.424331:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.424338:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.424341:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81518592, 81522688) 0 0 00000080:00200000:2.0:1789760795.424345:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.424357:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19902, 19902] 00020000:00200000:2.0:1789760795.424363:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81518592 chunk: [81518592, 81522688) 81522688, 4096 00020000:00200000:2.0:1789760795.424367:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.424381:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19902, 19902] 00000008:00010000:2.0:1789760795.424394:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.424399:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.424407:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81518592, 81522688) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.424418:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81518592=0x4dbe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.424463:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.424466:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81522688 00000020:00010000:2.0:1789760795.424472:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.424474:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.424477:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.424480:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.424483:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.424915:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.424924:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.424943:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c540 00000008:00100000:2.0:1789760795.425028:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26680 - object 0x280000401:2 offset 81518592<>81522688 00000100:00100000:3.0:1789760795.425116:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503937664:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.436398:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987209 gen 2 00000008:00010000:3.0:1789760795.436435:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760795.436441:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.436450:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.436511:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760795.436535:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503937664:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.439088:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19902, 19902] 00000080:00200000:2.0:1789760795.439106:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.439111:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.439124:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81522688 i_blocks 159248 00000020:00200000:2.0:1789760795.439128:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.439202:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.439227:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81522688, count: 4096 00000080:00200000:2.0:1789760795.439234:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81522688, bytes: 4096 00000080:00200000:2.0:1789760795.439245:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.439254:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.439256:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81522688, 81526784) 0 0 00000080:00200000:2.0:1789760795.439260:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.439272:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19903, 19903] 00020000:00200000:2.0:1789760795.439280:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81522688 chunk: [81522688, 81526784) 81526784, 4096 00020000:00200000:2.0:1789760795.439394:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.439416:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19903, 19903] 00000008:00010000:2.0:1789760795.439430:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.439435:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.439447:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81522688, 81526784) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.439460:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81522688=0x4dbf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.439775:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.439778:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81526784 00000020:00010000:2.0:1789760795.439787:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.439790:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.439793:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.439797:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.439800:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.439807:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.439813:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.439901:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c600 00000008:00100000:2.0:1789760795.440004:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25500 - object 0x280000401:2 offset 81522688<>81526784 00000100:00100000:3.0:1789760795.440623:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503937792:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.452962:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987210 gen 2 00000008:00010000:3.0:1789760795.453000:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760795.453006:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.453238:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.453252:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760795.453274:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503937792:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760795.453874:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19903, 19903] 00000080:00200000:1.0:1789760795.453889:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.453894:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.453907:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81526784 i_blocks 159256 00000020:00200000:1.0:1789760795.453911:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.453920:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.453943:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81526784, count: 4096 00000080:00200000:1.0:1789760795.453949:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81526784, bytes: 4096 00000080:00200000:1.0:1789760795.453957:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.453964:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760795.453967:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81526784, 81530880) 0 0 00000080:00200000:1.0:1789760795.453970:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.453982:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19904, 19904] 00020000:00200000:1.0:1789760795.453989:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81526784 chunk: [81526784, 81530880) 81530880, 4096 00020000:00200000:1.0:1789760795.453993:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.454006:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19904, 19904] 00000008:00010000:1.0:1789760795.454021:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.454026:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760795.454054:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81526784, 81530880) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.454065:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81526784=0x4dc0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.454113:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.454115:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81530880 00000020:00010000:1.0:1789760795.454122:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.454125:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.454128:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.454131:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.454135:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.454142:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.454148:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.454161:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760795.454244:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 81526784<>81530880 00000100:00100000:0.0:1789760795.454620:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503937920:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760795.469874:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987211 gen 2 00000008:00010000:0.0:1789760795.469912:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:0.0:1789760795.469917:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.469924:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.469944:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:0.0:1789760795.469965:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503937920:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760795.471380:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19904, 19904] 00000080:00200000:1.0:1789760795.471404:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.471410:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.471423:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81530880 i_blocks 159264 00000020:00200000:1.0:1789760795.471427:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.471437:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.471464:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81530880, count: 4096 00000080:00200000:1.0:1789760795.471471:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81530880, bytes: 4096 00000080:00200000:1.0:1789760795.471482:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.471489:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760795.471491:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81530880, 81534976) 0 0 00000080:00200000:1.0:1789760795.471495:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.471507:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19905, 19905] 00020000:00200000:1.0:1789760795.471513:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81530880 chunk: [81530880, 81534976) 81534976, 4096 00020000:00200000:1.0:1789760795.471517:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.471535:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19905, 19905] 00000008:00010000:1.0:1789760795.471550:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.471555:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760795.471573:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81530880, 81534976) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.471585:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81530880=0x4dc1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.471636:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.471639:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81534976 00000020:00010000:1.0:1789760795.471764:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.471769:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.471772:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.471776:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.471778:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.471787:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.471793:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.471812:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760795.471894:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 81530880<>81534976 00000100:00100000:0.0:1789760795.471969:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503938048:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760795.486148:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987212 gen 2 00000008:00010000:0.0:1789760795.486190:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58000 00000020:00200000:0.0:1789760795.486198:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.486208:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.486229:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58000 00000100:00100000:0.0:1789760795.486256:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503938048:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760795.486354:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19905, 19905] 00000080:00200000:1.0:1789760795.486369:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.486375:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.486388:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81534976 i_blocks 159272 00000020:00200000:1.0:1789760795.486391:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.486400:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.486428:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81534976, count: 4096 00000080:00200000:1.0:1789760795.486434:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81534976, bytes: 4096 00000080:00200000:1.0:1789760795.486441:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.486448:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760795.486450:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81534976, 81539072) 0 0 00000080:00200000:1.0:1789760795.486453:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.486466:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19906, 19906] 00020000:00200000:1.0:1789760795.486473:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81534976 chunk: [81534976, 81539072) 81539072, 4096 00020000:00200000:1.0:1789760795.486477:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.486493:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19906, 19906] 00000008:00010000:1.0:1789760795.486507:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.486513:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760795.486523:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81534976, 81539072) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.486535:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81534976=0x4dc2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.486587:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.486589:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81539072 00000020:00010000:1.0:1789760795.486595:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.486598:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.486601:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.486604:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.486606:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.486611:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.486616:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.486628:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760795.486704:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 81534976<>81539072 00000100:00100000:0.0:1789760795.486830:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503938176:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760795.498953:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987213 gen 2 00000008:00010000:0.0:1789760795.498991:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:0.0:1789760795.498998:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.499008:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.499247:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000100:00100000:0.0:1789760795.499273:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503938176:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760795.500923:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19906, 19906] 00000080:00200000:1.0:1789760795.500941:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.500947:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.500963:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81539072 i_blocks 159280 00000020:00200000:1.0:1789760795.500967:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.500977:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.501005:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81539072, count: 4096 00000080:00200000:1.0:1789760795.501010:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81539072, bytes: 4096 00000080:00200000:1.0:1789760795.501020:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.501027:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760795.501029:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81539072, 81543168) 0 0 00000080:00200000:1.0:1789760795.501057:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.501071:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19907, 19907] 00020000:00200000:1.0:1789760795.501078:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81539072 chunk: [81539072, 81543168) 81543168, 4096 00020000:00200000:1.0:1789760795.501081:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.501096:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19907, 19907] 00000008:00010000:1.0:1789760795.501110:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.501115:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760795.501124:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81539072, 81543168) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.501146:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81539072=0x4dc3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.501605:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.501609:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81543168 00000020:00010000:1.0:1789760795.501616:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.501620:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.501623:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.501626:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.501628:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.501635:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.501641:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.501726:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58780 00000008:00100000:1.0:1789760795.501807:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 81539072<>81543168 00000100:00100000:0.0:1789760795.501900:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760795.501918:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503938304:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760795.514369:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987214 gen 2 00000008:00010000:0.0:1789760795.514405:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58780 00000020:00200000:0.0:1789760795.514412:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.514420:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.514440:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58780 00000100:00100000:0.0:1789760795.514463:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503938304:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760795.514736:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19907, 19907] 00000080:00200000:1.0:1789760795.514748:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.514753:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.514764:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81543168 i_blocks 159288 00000020:00200000:1.0:1789760795.514768:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.514776:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.514800:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81543168, count: 4096 00000080:00200000:1.0:1789760795.514805:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81543168, bytes: 4096 00000080:00200000:1.0:1789760795.514812:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.514819:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760795.514821:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81543168, 81547264) 0 0 00000080:00200000:1.0:1789760795.514825:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.514837:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19908, 19908] 00020000:00200000:1.0:1789760795.514844:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81543168 chunk: [81543168, 81547264) 81547264, 4096 00020000:00200000:1.0:1789760795.514848:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.514862:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19908, 19908] 00000008:00010000:1.0:1789760795.514874:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.514879:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760795.514888:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81543168, 81547264) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.514898:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81543168=0x4dc4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.514941:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.514943:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81547264 00000020:00010000:1.0:1789760795.514949:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.514953:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.514955:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.514959:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.514961:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.514968:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.514974:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.514987:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760795.515095:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 81543168<>81547264 00000100:00100000:0.0:1789760795.515139:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503938432:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760795.529878:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987215 gen 2 00000008:00010000:0.0:1789760795.529913:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:0.0:1789760795.529919:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.529928:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.529951:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:0.0:1789760795.529975:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503938432:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760795.529994:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19908, 19908] 00000080:00200000:3.0:1789760795.530009:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.530015:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.530028:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81547264 i_blocks 159296 00000020:00200000:3.0:1789760795.530069:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.530079:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.530105:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81547264, count: 4096 00000080:00200000:3.0:1789760795.530112:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81547264, bytes: 4096 00000080:00200000:3.0:1789760795.530119:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.530127:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760795.530129:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81547264, 81551360) 0 0 00000080:00200000:3.0:1789760795.530133:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.530146:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19909, 19909] 00020000:00200000:3.0:1789760795.530153:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81547264 chunk: [81547264, 81551360) 81551360, 4096 00020000:00200000:3.0:1789760795.530156:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.530171:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19909, 19909] 00000008:00010000:3.0:1789760795.530185:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.530190:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760795.530199:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81547264, 81551360) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.530210:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81547264=0x4dc5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.530254:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.530258:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81551360 00000020:00010000:3.0:1789760795.530265:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.530268:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.530271:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.530274:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.530277:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.530298:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.530305:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.530318:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760795.530395:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070fb80 - object 0x280000401:2 offset 81547264<>81551360 00000100:00100000:3.0:1789760795.530434:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760795.530451:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503938560:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.541210:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987216 gen 2 00000008:00010000:3.0:1789760795.541244:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760795.541251:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.541258:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.541276:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760795.541297:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503938560:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.541474:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19909, 19909] 00000080:00200000:2.0:1789760795.541487:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.541504:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.541516:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81551360 i_blocks 159304 00000020:00200000:2.0:1789760795.541520:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.541528:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.541555:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81551360, count: 4096 00000080:00200000:2.0:1789760795.541577:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81551360, bytes: 4096 00000080:00200000:2.0:1789760795.541587:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.541594:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.541596:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81551360, 81555456) 0 0 00000080:00200000:2.0:1789760795.541852:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.541865:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19910, 19910] 00020000:00200000:2.0:1789760795.541873:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81551360 chunk: [81551360, 81555456) 81555456, 4096 00020000:00200000:2.0:1789760795.541877:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.541892:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19910, 19910] 00000008:00010000:2.0:1789760795.541906:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.541911:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.541921:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81551360, 81555456) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.541932:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81551360=0x4dc6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.541978:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.541981:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81555456 00000020:00010000:2.0:1789760795.541988:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.541991:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.541994:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.541997:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.542000:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.542007:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.542012:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.542027:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cb40 00000008:00100000:2.0:1789760795.542140:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24000 - object 0x280000401:2 offset 81551360<>81555456 00000100:00100000:2.0:1789760795.542175:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760795.542190:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503938688:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.557987:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987217 gen 2 00000008:00010000:3.0:1789760795.558021:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760795.558027:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.558071:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.558088:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760795.558110:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503938688:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.558195:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19910, 19910] 00000080:00200000:2.0:1789760795.558207:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.558213:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.558224:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81555456 i_blocks 159312 00000020:00200000:2.0:1789760795.558227:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.558234:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.558257:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81555456, count: 4096 00000080:00200000:2.0:1789760795.558262:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81555456, bytes: 4096 00000080:00200000:2.0:1789760795.558269:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.558276:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.558278:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81555456, 81559552) 0 0 00000080:00200000:2.0:1789760795.558282:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.558294:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19911, 19911] 00020000:00200000:2.0:1789760795.558301:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81555456 chunk: [81555456, 81559552) 81559552, 4096 00020000:00200000:2.0:1789760795.558305:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.558318:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19911, 19911] 00000008:00010000:2.0:1789760795.558331:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.558337:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.558346:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81555456, 81559552) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.558357:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81555456=0x4dc7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.558398:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.558401:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81559552 00000020:00010000:2.0:1789760795.558409:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.558412:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.558414:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.558417:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.558420:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.558426:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.558432:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.558446:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000008:00100000:2.0:1789760795.558752:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24000 - object 0x280000401:2 offset 81555456<>81559552 00000100:00100000:3.0:1789760795.558813:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503938816:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.579269:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987218 gen 2 00000008:00010000:3.0:1789760795.579317:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760795.579323:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.579332:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.579351:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760795.579375:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503938816:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.579832:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19911, 19911] 00000080:00200000:2.0:1789760795.579845:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.579850:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.579861:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81559552 i_blocks 159320 00000020:00200000:2.0:1789760795.579865:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.579874:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.579908:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81559552, count: 4096 00000080:00200000:2.0:1789760795.579915:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81559552, bytes: 4096 00000080:00200000:2.0:1789760795.579924:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.579931:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.579933:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81559552, 81563648) 0 0 00000080:00200000:2.0:1789760795.579936:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.579949:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19912, 19912] 00020000:00200000:2.0:1789760795.579956:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81559552 chunk: [81559552, 81563648) 81563648, 4096 00020000:00200000:2.0:1789760795.579960:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.579975:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19912, 19912] 00000008:00010000:2.0:1789760795.579989:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.579994:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.580003:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81559552, 81563648) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.580013:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81559552=0x4dc8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.580112:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.580115:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81563648 00000020:00010000:2.0:1789760795.580122:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.580125:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.580128:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.580131:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.580133:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.580140:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.580145:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.580155:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c900 00000008:00100000:2.0:1789760795.580233:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25500 - object 0x280000401:2 offset 81559552<>81563648 00000100:00100000:3.0:1789760795.580456:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503938944:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.588855:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987219 gen 2 00000008:00010000:3.0:1789760795.588893:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760795.588911:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.588920:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.588983:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760795.589008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503938944:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.589399:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19912, 19912] 00000080:00200000:2.0:1789760795.589414:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.589418:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.589430:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81563648 i_blocks 159328 00000020:00200000:2.0:1789760795.589433:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.589441:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.589467:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81563648, count: 4096 00000080:00200000:2.0:1789760795.589473:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81563648, bytes: 4096 00000080:00200000:2.0:1789760795.589482:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.589489:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.589491:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81563648, 81567744) 0 0 00000080:00200000:2.0:1789760795.589495:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.589508:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19913, 19913] 00020000:00200000:2.0:1789760795.589515:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81563648 chunk: [81563648, 81567744) 81567744, 4096 00020000:00200000:2.0:1789760795.589518:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.589533:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19913, 19913] 00000008:00010000:2.0:1789760795.589546:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.589552:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.589570:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81563648, 81567744) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.589581:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81563648=0x4dc9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.589752:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.589755:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81567744 00000020:00010000:2.0:1789760795.589763:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.589766:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.589770:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.589773:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.589775:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.589784:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.589790:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.589806:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c240 00000008:00100000:2.0:1789760795.589878:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26680 - object 0x280000401:2 offset 81563648<>81567744 00000100:00100000:3.0:1789760795.590006:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503939072:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.598152:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987220 gen 2 00000008:00010000:3.0:1789760795.598188:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760795.598195:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.598203:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.598223:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760795.598245:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503939072:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.598339:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19913, 19913] 00000080:00200000:2.0:1789760795.598352:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.598357:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.598368:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81567744 i_blocks 159336 00000020:00200000:2.0:1789760795.598371:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.598381:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.598404:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81567744, count: 4096 00000080:00200000:2.0:1789760795.598410:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81567744, bytes: 4096 00000080:00200000:2.0:1789760795.598418:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.598425:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.598426:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81567744, 81571840) 0 0 00000080:00200000:2.0:1789760795.598429:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.598440:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19914, 19914] 00020000:00200000:2.0:1789760795.598446:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81567744 chunk: [81567744, 81571840) 81571840, 4096 00020000:00200000:2.0:1789760795.598450:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.598464:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19914, 19914] 00000008:00010000:2.0:1789760795.598477:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.598483:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.598490:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81567744, 81571840) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.598500:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81567744=0x4dca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.598541:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.598543:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81571840 00000020:00010000:2.0:1789760795.598550:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.598553:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.598556:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.598568:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.598572:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.598579:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.598585:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.598608:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760795.598696:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27480 - object 0x280000401:2 offset 81567744<>81571840 00000100:00100000:3.0:1789760795.598968:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503939200:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.609352:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987221 gen 2 00000008:00010000:3.0:1789760795.609390:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760795.609396:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.609404:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.609426:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760795.609449:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503939200:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.609686:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19914, 19914] 00000080:00200000:2.0:1789760795.609699:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.609703:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.609716:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81571840 i_blocks 159344 00000020:00200000:2.0:1789760795.609720:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.609727:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.609754:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81571840, count: 4096 00000080:00200000:2.0:1789760795.609759:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81571840, bytes: 4096 00000080:00200000:2.0:1789760795.609766:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.609773:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.609775:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81571840, 81575936) 0 0 00000080:00200000:2.0:1789760795.609778:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.609789:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19915, 19915] 00020000:00200000:2.0:1789760795.609795:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81571840 chunk: [81571840, 81575936) 81575936, 4096 00020000:00200000:2.0:1789760795.609799:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.609812:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19915, 19915] 00000008:00010000:2.0:1789760795.609825:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.609830:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.609838:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81571840, 81575936) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.609848:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81571840=0x4dcb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.609889:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.609891:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81575936 00000020:00010000:2.0:1789760795.610767:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.610773:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.610777:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.610780:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.610783:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.610793:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.610801:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.610819:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ce40 00000008:00100000:2.0:1789760795.610893:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24e00 - object 0x280000401:2 offset 81571840<>81575936 00000100:00100000:3.0:1789760795.611746:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760795.611766:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503939328:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.624393:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987222 gen 2 00000008:00010000:3.0:1789760795.624431:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760795.624436:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.624444:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.624462:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760795.624485:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503939328:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.624581:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19915, 19915] 00000080:00200000:2.0:1789760795.624593:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.624598:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.624618:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81575936 i_blocks 159352 00000020:00200000:2.0:1789760795.624622:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.624630:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.624654:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81575936, count: 4096 00000080:00200000:2.0:1789760795.624660:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81575936, bytes: 4096 00000080:00200000:2.0:1789760795.624667:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.624674:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.624676:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81575936, 81580032) 0 0 00000080:00200000:2.0:1789760795.624680:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.624692:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19916, 19916] 00020000:00200000:2.0:1789760795.624698:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81575936 chunk: [81575936, 81580032) 81580032, 4096 00020000:00200000:2.0:1789760795.624702:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.624717:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19916, 19916] 00000008:00010000:2.0:1789760795.624730:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.624736:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.624744:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81575936, 81580032) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.624755:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81575936=0x4dcc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.624796:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.624799:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81580032 00000020:00010000:2.0:1789760795.624805:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.624808:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.624811:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.624815:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.624817:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.624823:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.624829:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.624843:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c000 00000008:00100000:2.0:1789760795.624924:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 81575936<>81580032 00000100:00100000:3.0:1789760795.624975:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503939456:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.638297:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987223 gen 2 00000008:00010000:3.0:1789760795.638441:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760795.638454:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.638462:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.638481:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760795.638504:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503939456:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.638594:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19916, 19916] 00000080:00200000:2.0:1789760795.638617:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.638622:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.638635:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81580032 i_blocks 159360 00000020:00200000:2.0:1789760795.638639:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.638646:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.638672:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81580032, count: 4096 00000080:00200000:2.0:1789760795.638677:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81580032, bytes: 4096 00000080:00200000:2.0:1789760795.638682:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.638698:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.638700:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81580032, 81584128) 0 0 00000080:00200000:2.0:1789760795.638705:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.638718:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19917, 19917] 00020000:00200000:2.0:1789760795.638725:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81580032 chunk: [81580032, 81584128) 81584128, 4096 00020000:00200000:2.0:1789760795.638729:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.638749:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19917, 19917] 00000008:00010000:2.0:1789760795.638764:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.638769:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.638779:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81580032, 81584128) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.638800:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81580032=0x4dcd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.638844:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.638847:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81584128 00000020:00010000:2.0:1789760795.638853:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.638856:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.638859:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.638862:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.638865:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.638872:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.638878:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.638892:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cf00 00000008:00100000:2.0:1789760795.638970:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24700 - object 0x280000401:2 offset 81580032<>81584128 00000100:00100000:3.0:1789760795.639408:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503939584:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.650657:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987224 gen 2 00000008:00010000:3.0:1789760795.650696:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760795.650703:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.650712:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.650734:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760795.650761:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503939584:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.650835:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19917, 19917] 00000080:00200000:2.0:1789760795.650848:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.650853:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.650866:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81584128 i_blocks 159368 00000020:00200000:2.0:1789760795.650869:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.650877:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.650913:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81584128, count: 4096 00000080:00200000:2.0:1789760795.650920:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81584128, bytes: 4096 00000080:00200000:2.0:1789760795.650932:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.650942:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.650944:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81584128, 81588224) 0 0 00000080:00200000:2.0:1789760795.650948:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.650961:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19918, 19918] 00020000:00200000:2.0:1789760795.650968:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81584128 chunk: [81584128, 81588224) 81588224, 4096 00020000:00200000:2.0:1789760795.650971:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.650985:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19918, 19918] 00000008:00010000:2.0:1789760795.650999:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.651004:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.651014:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81584128, 81588224) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.651024:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81584128=0x4dce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.651087:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.651090:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81588224 00000020:00010000:2.0:1789760795.651097:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.651100:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.651103:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.651106:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.651108:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.651115:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.651120:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.651134:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cc00 00000008:00100000:2.0:1789760795.651208:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24380 - object 0x280000401:2 offset 81584128<>81588224 00000100:00100000:3.0:1789760795.651468:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760795.651482:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503939712:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.666923:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987225 gen 2 00000008:00010000:3.0:1789760795.666962:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760795.666968:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.666977:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.666998:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760795.667027:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503939712:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.668263:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19918, 19918] 00000080:00200000:2.0:1789760795.668279:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.668284:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.668309:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81588224 i_blocks 159376 00000020:00200000:2.0:1789760795.668314:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.668323:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.668348:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81588224, count: 4096 00000080:00200000:2.0:1789760795.668353:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81588224, bytes: 4096 00000080:00200000:2.0:1789760795.668361:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.668368:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.668370:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81588224, 81592320) 0 0 00000080:00200000:2.0:1789760795.668374:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.668386:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19919, 19919] 00020000:00200000:2.0:1789760795.668392:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81588224 chunk: [81588224, 81592320) 81592320, 4096 00020000:00200000:2.0:1789760795.668396:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.668411:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19919, 19919] 00000008:00010000:2.0:1789760795.668423:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.668428:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.668436:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81588224, 81592320) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.668446:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81588224=0x4dcf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.668493:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.668495:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81592320 00000020:00010000:2.0:1789760795.668501:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.668504:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.668507:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.668510:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.668513:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.668520:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.668525:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.668540:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c300 00000008:00100000:2.0:1789760795.668872:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25c00 - object 0x280000401:2 offset 81588224<>81592320 00000100:00100000:2.0:1789760795.668918:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760795.668936:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503939840:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.681452:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987226 gen 2 00000008:00010000:3.0:1789760795.681487:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760795.681493:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.681502:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.681523:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760795.681546:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503939840:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.681656:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19919, 19919] 00000080:00200000:2.0:1789760795.681668:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.681673:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.681684:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81592320 i_blocks 159384 00000020:00200000:2.0:1789760795.681688:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.681706:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.681731:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81592320, count: 4096 00000080:00200000:2.0:1789760795.681738:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81592320, bytes: 4096 00000080:00200000:2.0:1789760795.681745:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.681751:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.681753:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81592320, 81596416) 0 0 00000080:00200000:2.0:1789760795.681757:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.681769:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19920, 19920] 00020000:00200000:2.0:1789760795.681776:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81592320 chunk: [81592320, 81596416) 81596416, 4096 00020000:00200000:2.0:1789760795.681780:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.681795:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19920, 19920] 00000008:00010000:2.0:1789760795.681807:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.681812:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.681820:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81592320, 81596416) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.681831:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81592320=0x4dd0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.681872:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.681875:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81596416 00000020:00010000:2.0:1789760795.681881:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.681885:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.681887:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.681890:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.681893:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.681899:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.681904:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.681918:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000008:00100000:2.0:1789760795.681990:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25180 - object 0x280000401:2 offset 81592320<>81596416 00000100:00100000:3.0:1789760795.682165:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503939968:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.695631:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987227 gen 2 00000008:00010000:2.0:1789760795.695756:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:2.0:1789760795.695764:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.695773:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.695792:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:2.0:1789760795.695816:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503939968:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760795.695834:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19920, 19920] 00000080:00200000:0.0:1789760795.695849:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.695854:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.695867:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81596416 i_blocks 159392 00000020:00200000:0.0:1789760795.695871:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.695880:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.695904:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81596416, count: 4096 00000080:00200000:0.0:1789760795.695911:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81596416, bytes: 4096 00000080:00200000:0.0:1789760795.695928:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.695936:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760795.695938:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81596416, 81600512) 0 0 00000080:00200000:0.0:1789760795.695941:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.695955:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19921, 19921] 00020000:00200000:0.0:1789760795.695962:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81596416 chunk: [81596416, 81600512) 81600512, 4096 00020000:00200000:0.0:1789760795.695966:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.695981:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19921, 19921] 00000008:00010000:0.0:1789760795.695995:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.696000:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760795.696008:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81596416, 81600512) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.696029:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81596416=0x4dd1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.696097:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.696100:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81600512 00000020:00010000:0.0:1789760795.696107:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.696120:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.696122:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.696125:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.696128:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.696135:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.696141:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.696154:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760795.696580:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 81596416<>81600512 00000100:00100000:0.0:1789760795.696620:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760795.696634:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503940096:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760795.704003:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987228 gen 2 00000008:00010000:0.0:1789760795.704062:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:0.0:1789760795.704069:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.704078:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.704595:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:0.0:1789760795.704619:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503940096:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760795.704691:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19921, 19921] 00000080:00200000:1.0:1789760795.704702:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.704708:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.704720:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81600512 i_blocks 159400 00000020:00200000:1.0:1789760795.704723:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.704731:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.704757:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81600512, count: 4096 00000080:00200000:1.0:1789760795.704762:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81600512, bytes: 4096 00000080:00200000:1.0:1789760795.704771:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.704779:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760795.704781:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81600512, 81604608) 0 0 00000080:00200000:1.0:1789760795.704784:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.704797:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19922, 19922] 00020000:00200000:1.0:1789760795.704803:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81600512 chunk: [81600512, 81604608) 81604608, 4096 00020000:00200000:1.0:1789760795.704806:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.704820:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19922, 19922] 00000008:00010000:1.0:1789760795.704833:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.704896:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760795.704906:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81600512, 81604608) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.704917:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81600512=0x4dd2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.704962:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.704964:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81604608 00000020:00010000:1.0:1789760795.704971:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.704975:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.704977:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.704981:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.704984:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.704990:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.704995:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.705008:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760795.705175:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81600512<>81604608 00000100:00100000:1.0:1789760795.705211:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760795.705225:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503940224:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760795.712423:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987229 gen 2 00000008:00010000:1.0:1789760795.712461:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:1.0:1789760795.712467:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.712476:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.712496:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:1.0:1789760795.712520:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503940224:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760795.712703:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19922, 19922] 00000080:00200000:0.0:1789760795.712716:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.712720:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.712733:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81604608 i_blocks 159408 00000020:00200000:0.0:1789760795.712736:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.712745:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.712772:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81604608, count: 4096 00000080:00200000:0.0:1789760795.712778:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81604608, bytes: 4096 00000080:00200000:0.0:1789760795.712787:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.712793:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760795.712796:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81604608, 81608704) 0 0 00000080:00200000:0.0:1789760795.712799:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.712813:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19923, 19923] 00020000:00200000:0.0:1789760795.712820:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81604608 chunk: [81604608, 81608704) 81608704, 4096 00020000:00200000:0.0:1789760795.712886:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.712903:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19923, 19923] 00000008:00010000:0.0:1789760795.712916:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.712922:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760795.712931:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81604608, 81608704) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.712940:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81604608=0x4dd3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.712974:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.712975:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81608704 00000020:00010000:0.0:1789760795.712981:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.712983:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.712985:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.712987:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.712989:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.712994:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.712997:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.713008:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760795.713100:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 81604608<>81608704 00000100:00100000:0.0:1789760795.713133:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760795.713145:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503940352:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760795.722259:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987230 gen 2 00000008:00010000:0.0:1789760795.722377:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:0.0:1789760795.722383:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.722393:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.722458:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:0.0:1789760795.722482:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503940352:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.722760:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19923, 19923] 00000080:00200000:2.0:1789760795.723459:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.723466:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.723479:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81608704 i_blocks 159416 00000020:00200000:2.0:1789760795.723483:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.723493:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.723517:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81608704, count: 4096 00000080:00200000:2.0:1789760795.723524:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81608704, bytes: 4096 00000080:00200000:2.0:1789760795.723534:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.723542:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.723543:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81608704, 81612800) 0 0 00000080:00200000:2.0:1789760795.723547:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.723569:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19924, 19924] 00020000:00200000:2.0:1789760795.723577:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81608704 chunk: [81608704, 81612800) 81612800, 4096 00020000:00200000:2.0:1789760795.723580:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.723596:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19924, 19924] 00000008:00010000:2.0:1789760795.723682:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.723757:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.723768:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81608704, 81612800) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.723780:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81608704=0x4dd4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.723827:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.723829:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81612800 00000020:00010000:2.0:1789760795.723836:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.723839:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.723841:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.723844:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.723846:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.723852:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.724234:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.724257:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000008:00100000:2.0:1789760795.724403:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25180 - object 0x280000401:2 offset 81608704<>81612800 00000100:00100000:3.0:1789760795.724476:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503940480:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.735397:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987231 gen 2 00000008:00010000:3.0:1789760795.735431:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760795.735437:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.735444:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.735464:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760795.735554:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503940480:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.735991:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19924, 19924] 00000080:00200000:2.0:1789760795.736005:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.736011:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.736023:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81612800 i_blocks 159424 00000020:00200000:2.0:1789760795.736027:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.736072:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.736099:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81612800, count: 4096 00000080:00200000:2.0:1789760795.736105:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81612800, bytes: 4096 00000080:00200000:2.0:1789760795.736113:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.736121:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.736123:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81612800, 81616896) 0 0 00000080:00200000:2.0:1789760795.736127:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.736139:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19925, 19925] 00020000:00200000:2.0:1789760795.736147:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81612800 chunk: [81612800, 81616896) 81616896, 4096 00020000:00200000:2.0:1789760795.736152:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.736167:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19925, 19925] 00000008:00010000:2.0:1789760795.736181:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.736186:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.736270:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81612800, 81616896) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.736282:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81612800=0x4dd5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.736328:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.736331:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81616896 00000020:00010000:2.0:1789760795.736338:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.736342:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.736345:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.736348:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.736350:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.736358:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.736363:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.736376:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000008:00100000:2.0:1789760795.736456:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27800 - object 0x280000401:2 offset 81612800<>81616896 00000100:00100000:3.0:1789760795.736526:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503940608:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.748365:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987232 gen 2 00000008:00010000:3.0:1789760795.748403:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760795.748409:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.748418:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.748441:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760795.748465:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503940608:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.748498:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19925, 19925] 00000080:00200000:2.0:1789760795.748514:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.748519:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.748531:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81616896 i_blocks 159432 00000020:00200000:2.0:1789760795.748535:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.748543:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.748579:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81616896, count: 4096 00000080:00200000:2.0:1789760795.748586:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81616896, bytes: 4096 00000080:00200000:2.0:1789760795.748595:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.748603:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.748605:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81616896, 81620992) 0 0 00000080:00200000:2.0:1789760795.748609:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.748621:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19926, 19926] 00020000:00200000:2.0:1789760795.748628:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81616896 chunk: [81616896, 81620992) 81620992, 4096 00020000:00200000:2.0:1789760795.748632:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.748645:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19926, 19926] 00000008:00010000:2.0:1789760795.748659:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.748664:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.748673:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81616896, 81620992) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.748684:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81616896=0x4dd6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.748729:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.748731:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81620992 00000020:00010000:2.0:1789760795.748738:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.748742:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.748744:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.748747:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.748750:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.748756:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.748762:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.748776:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c480 00000008:00100000:2.0:1789760795.748850:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26a00 - object 0x280000401:2 offset 81616896<>81620992 00000100:00100000:3.0:1789760795.748922:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503940736:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.766360:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987233 gen 2 00000008:00010000:3.0:1789760795.766406:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760795.766414:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.766426:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.766454:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760795.766485:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503940736:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.766612:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19926, 19926] 00000080:00200000:2.0:1789760795.766624:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.766629:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.766640:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81620992 i_blocks 159440 00000020:00200000:2.0:1789760795.766642:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.766650:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.766674:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81620992, count: 4096 00000080:00200000:2.0:1789760795.766680:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81620992, bytes: 4096 00000080:00200000:2.0:1789760795.766692:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.766699:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.766701:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81620992, 81625088) 0 0 00000080:00200000:2.0:1789760795.766705:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.766718:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19927, 19927] 00020000:00200000:2.0:1789760795.766725:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81620992 chunk: [81620992, 81625088) 81625088, 4096 00020000:00200000:2.0:1789760795.766730:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.766746:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19927, 19927] 00000008:00010000:2.0:1789760795.766759:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.766764:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.766772:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81620992, 81625088) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.766783:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81620992=0x4dd7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.766830:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.766833:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81625088 00000020:00010000:2.0:1789760795.766839:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.766842:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.766845:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.766849:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.766851:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.766870:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.766876:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.766888:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c480 00000008:00100000:2.0:1789760795.766977:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25f80 - object 0x280000401:2 offset 81620992<>81625088 00000100:00100000:3.0:1789760795.767190:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503940864:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.777578:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987234 gen 2 00000008:00010000:3.0:1789760795.777616:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760795.777687:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.777696:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.777716:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760795.777741:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503940864:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.779301:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19927, 19927] 00000080:00200000:2.0:1789760795.779319:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.779325:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.779339:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81625088 i_blocks 159448 00000020:00200000:2.0:1789760795.779343:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.779353:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.779379:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81625088, count: 4096 00000080:00200000:2.0:1789760795.779386:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81625088, bytes: 4096 00000080:00200000:2.0:1789760795.779396:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.779403:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.779405:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81625088, 81629184) 0 0 00000080:00200000:2.0:1789760795.779409:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.779421:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19928, 19928] 00020000:00200000:2.0:1789760795.779427:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81625088 chunk: [81625088, 81629184) 81629184, 4096 00020000:00200000:2.0:1789760795.779430:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.779445:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19928, 19928] 00000008:00010000:2.0:1789760795.779458:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.779462:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.779471:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81625088, 81629184) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.779482:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81625088=0x4dd8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.779533:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.779536:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81629184 00000020:00010000:2.0:1789760795.779543:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.779546:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.779549:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.779551:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.779554:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.779573:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.779580:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.779595:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000008:00100000:2.0:1789760795.779680:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26300 - object 0x280000401:2 offset 81625088<>81629184 00000100:00100000:3.0:1789760795.780127:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760795.780147:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503940992:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.796243:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987235 gen 2 00000008:00010000:3.0:1789760795.796278:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760795.796284:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.796293:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.796308:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760795.796327:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503940992:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.798587:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19928, 19928] 00000080:00200000:2.0:1789760795.798922:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.798928:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.798942:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81629184 i_blocks 159456 00000020:00200000:2.0:1789760795.798947:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.798957:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.798982:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81629184, count: 4096 00000080:00200000:2.0:1789760795.798989:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81629184, bytes: 4096 00000080:00200000:2.0:1789760795.798999:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.799007:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.799009:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81629184, 81633280) 0 0 00000080:00200000:2.0:1789760795.799013:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.799026:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19929, 19929] 00020000:00200000:2.0:1789760795.799760:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81629184 chunk: [81629184, 81633280) 81633280, 4096 00020000:00200000:2.0:1789760795.799767:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.799787:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19929, 19929] 00000008:00010000:2.0:1789760795.799802:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.799807:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.799816:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81629184, 81633280) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.799827:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81629184=0x4dd9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.799875:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.799877:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81633280 00000020:00010000:2.0:1789760795.799883:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.799886:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.799888:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.799891:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.799894:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.799901:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.799907:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.799928:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000008:00100000:2.0:1789760795.800240:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24a80 - object 0x280000401:2 offset 81629184<>81633280 00000100:00100000:3.0:1789760795.800324:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503941120:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.814582:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987236 gen 2 00000008:00010000:3.0:1789760795.814622:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760795.814629:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.814637:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.814654:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760795.814678:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503941120:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.814923:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19929, 19929] 00000080:00200000:2.0:1789760795.814937:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.814942:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.814953:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81633280 i_blocks 159464 00000020:00200000:2.0:1789760795.814957:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.814964:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.814990:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81633280, count: 4096 00000080:00200000:2.0:1789760795.814995:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81633280, bytes: 4096 00000080:00200000:2.0:1789760795.815002:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.815010:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.815012:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81633280, 81637376) 0 0 00000080:00200000:2.0:1789760795.815016:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.815028:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19930, 19930] 00020000:00200000:2.0:1789760795.815076:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81633280 chunk: [81633280, 81637376) 81637376, 4096 00020000:00200000:2.0:1789760795.815082:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.815097:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19930, 19930] 00000008:00010000:2.0:1789760795.815112:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.815117:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.815126:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81633280, 81637376) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.815136:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81633280=0x4dda000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.815189:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.815192:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81637376 00000020:00010000:2.0:1789760795.815199:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.815202:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.815205:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.815208:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.815211:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.815217:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.815222:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.815235:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c300 00000008:00100000:2.0:1789760795.815311:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27100 - object 0x280000401:2 offset 81633280<>81637376 00000100:00100000:3.0:1789760795.815512:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503941248:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.833644:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987237 gen 2 00000008:00010000:3.0:1789760795.833681:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760795.833806:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.833816:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.833833:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000080:00200000:2.0:1789760795.833877:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19930, 19930] 00000080:00200000:2.0:1789760795.833892:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.833897:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.833909:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81637376 i_blocks 159472 00000020:00200000:2.0:1789760795.833913:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.833921:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.833947:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81637376, count: 4096 00000080:00200000:2.0:1789760795.833953:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81637376, bytes: 4096 00000080:00200000:2.0:1789760795.833962:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.833969:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.833971:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81637376, 81641472) 0 0 00000080:00200000:2.0:1789760795.833975:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.833988:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19931, 19931] 00020000:00200000:2.0:1789760795.833995:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81637376 chunk: [81637376, 81641472) 81641472, 4096 00020000:00200000:2.0:1789760795.834000:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.834015:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19931, 19931] 00000008:00010000:2.0:1789760795.834026:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.834030:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.834233:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81637376, 81641472) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.834244:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81637376=0x4ddb000, pages 1 (max 131072), lockless, parallel 00000100:00100000:3.0:1789760795.834316:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503941248:192.168.203.158@tcp:4:dd.0 00000020:00200000:2.0:1789760795.834326:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.834329:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81641472 00000020:00010000:2.0:1789760795.834335:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.834338:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.834340:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.834343:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.834344:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.834350:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.834355:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.834356:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cc00 00000008:00100000:3.0:1789760795.834431:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070fb80 - object 0x280000401:2 offset 81637376<>81641472 00000100:00100000:3.0:1789760795.834464:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503941376:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.866497:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987238 gen 2 00000008:00010000:3.0:1789760795.866535:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760795.866542:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.866551:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.866588:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760795.866617:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503941376:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.866778:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19931, 19931] 00000080:00200000:2.0:1789760795.866793:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.866798:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.866812:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81641472 i_blocks 159480 00000020:00200000:2.0:1789760795.866816:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.866824:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.866852:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81641472, count: 4096 00000080:00200000:2.0:1789760795.866858:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81641472, bytes: 4096 00000080:00200000:2.0:1789760795.866870:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.866877:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.866880:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81641472, 81645568) 0 0 00000080:00200000:2.0:1789760795.866884:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.866898:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19932, 19932] 00020000:00200000:2.0:1789760795.866905:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81641472 chunk: [81641472, 81645568) 81645568, 4096 00020000:00200000:2.0:1789760795.866910:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.866936:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19932, 19932] 00000008:00010000:2.0:1789760795.866950:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.866955:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.866965:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81641472, 81645568) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.866976:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81641472=0x4ddc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.867022:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.867025:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81645568 00000020:00010000:2.0:1789760795.867031:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.867056:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.867059:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.867063:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.867065:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.867261:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.867269:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.867287:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cf00 00000008:00100000:2.0:1789760795.867364:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27100 - object 0x280000401:2 offset 81641472<>81645568 00000100:00100000:3.0:1789760795.867469:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503941504:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.890953:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987239 gen 2 00000008:00010000:3.0:1789760795.890990:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760795.890996:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.891004:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.891024:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760795.891087:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503941504:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.891415:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19932, 19932] 00000080:00200000:2.0:1789760795.891429:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.891434:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.891447:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81645568 i_blocks 159488 00000020:00200000:2.0:1789760795.891450:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.891459:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.891486:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81645568, count: 4096 00000080:00200000:2.0:1789760795.891492:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81645568, bytes: 4096 00000080:00200000:2.0:1789760795.891500:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.891507:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.891509:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81645568, 81649664) 0 0 00000080:00200000:2.0:1789760795.891513:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.891526:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19933, 19933] 00020000:00200000:2.0:1789760795.891532:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81645568 chunk: [81645568, 81649664) 81649664, 4096 00020000:00200000:2.0:1789760795.891536:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.891552:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19933, 19933] 00000008:00010000:2.0:1789760795.891576:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.891581:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.891589:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81645568, 81649664) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.891599:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81645568=0x4ddd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.891649:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.891652:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81649664 00000020:00010000:2.0:1789760795.891659:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.891662:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.891664:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.891668:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.891670:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.891676:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.891682:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.891695:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c000 00000008:00100000:2.0:1789760795.891778:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24a80 - object 0x280000401:2 offset 81645568<>81649664 00000100:00100000:3.0:1789760795.893638:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503941632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.909742:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987240 gen 2 00000008:00010000:3.0:1789760795.909800:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760795.909863:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.909877:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.909899:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760795.909931:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503941632:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.910081:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19933, 19933] 00000080:00200000:2.0:1789760795.910101:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.910107:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.910132:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81649664 i_blocks 159496 00000020:00200000:2.0:1789760795.910136:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.910148:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.910186:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81649664, count: 4096 00000080:00200000:2.0:1789760795.910193:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81649664, bytes: 4096 00000080:00200000:2.0:1789760795.910202:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.910211:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.910213:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81649664, 81653760) 0 0 00000080:00200000:2.0:1789760795.910218:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.910235:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19934, 19934] 00020000:00200000:2.0:1789760795.910244:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81649664 chunk: [81649664, 81653760) 81653760, 4096 00020000:00200000:2.0:1789760795.910249:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.910269:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19934, 19934] 00000008:00010000:2.0:1789760795.910288:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.910294:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.910306:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81649664, 81653760) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.910323:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81649664=0x4dde000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.910386:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.910388:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81653760 00000020:00010000:2.0:1789760795.910396:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.910400:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.910403:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.910406:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.910409:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.910418:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.910424:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.910441:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ce40 00000008:00100000:2.0:1789760795.910574:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26300 - object 0x280000401:2 offset 81649664<>81653760 00000100:00100000:3.0:1789760795.911104:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503941760:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.929295:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987241 gen 2 00000008:00010000:3.0:1789760795.929351:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760795.929361:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.929374:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.929398:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760795.929430:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503941760:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.929536:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19934, 19934] 00000080:00200000:2.0:1789760795.929555:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.929572:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.929591:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81653760 i_blocks 159504 00000020:00200000:2.0:1789760795.929595:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.929605:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.929644:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81653760, count: 4096 00000080:00200000:2.0:1789760795.929651:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81653760, bytes: 4096 00000080:00200000:2.0:1789760795.929661:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.929669:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.929672:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81653760, 81657856) 0 0 00000080:00200000:2.0:1789760795.929677:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.929695:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19935, 19935] 00020000:00200000:2.0:1789760795.929704:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81653760 chunk: [81653760, 81657856) 81657856, 4096 00020000:00200000:2.0:1789760795.929709:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.929728:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19935, 19935] 00000008:00010000:2.0:1789760795.929748:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.929754:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.929766:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81653760, 81657856) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.929782:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81653760=0x4ddf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.929845:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.929848:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81657856 00000020:00010000:2.0:1789760795.929856:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.929860:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.929863:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.929866:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.929869:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.929879:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.929885:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.929901:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760795.930016:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25f80 - object 0x280000401:2 offset 81653760<>81657856 00000100:00100000:3.0:1789760795.930310:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503941888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.941691:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987242 gen 2 00000008:00010000:3.0:1789760795.941730:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760795.941737:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.941747:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.941766:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760795.941794:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503941888:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.941811:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19935, 19935] 00000080:00200000:2.0:1789760795.941834:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.941842:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.941862:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81657856 i_blocks 159512 00000020:00200000:2.0:1789760795.941868:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.941882:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.941935:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81657856, count: 4096 00000080:00200000:2.0:1789760795.941944:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81657856, bytes: 4096 00000080:00200000:2.0:1789760795.941953:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.941962:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.941965:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81657856, 81661952) 0 0 00000080:00200000:2.0:1789760795.941969:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.941984:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19936, 19936] 00020000:00200000:2.0:1789760795.941992:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81657856 chunk: [81657856, 81661952) 81661952, 4096 00020000:00200000:2.0:1789760795.941996:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.942017:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19936, 19936] 00000008:00010000:2.0:1789760795.942079:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.942086:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.942099:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81657856, 81661952) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.942112:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81657856=0x4de0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.942173:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.942176:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81661952 00000020:00010000:2.0:1789760795.942184:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.942188:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.942191:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.942194:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.942197:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.942209:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.942217:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.942238:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c240 00000008:00100000:2.0:1789760795.942350:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26a00 - object 0x280000401:2 offset 81657856<>81661952 00000100:00100000:3.0:1789760795.942429:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760795.942453:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503942016:192.168.203.158@tcp:4:dd.0 00010000:00010000:0.0:1789760795.967433:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760795.968475:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987243 gen 2 00000008:00010000:3.0:1789760795.968522:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760795.968531:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.968543:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.968578:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760795.968612:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503942016:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.968791:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19936, 19936] 00000080:00200000:2.0:1789760795.968811:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.968818:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.968836:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81661952 i_blocks 159520 00000020:00200000:2.0:1789760795.968840:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.968850:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.968889:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81661952, count: 4096 00000080:00200000:2.0:1789760795.968896:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81661952, bytes: 4096 00000080:00200000:2.0:1789760795.968912:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.968921:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.968924:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81661952, 81666048) 0 0 00000080:00200000:2.0:1789760795.968928:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.968945:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19937, 19937] 00020000:00200000:2.0:1789760795.968954:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81661952 chunk: [81661952, 81666048) 81666048, 4096 00020000:00200000:2.0:1789760795.968958:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.968978:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19937, 19937] 00000008:00010000:2.0:1789760795.968996:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.969001:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.969012:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81661952, 81666048) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.969025:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81661952=0x4de1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.969131:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.969134:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81666048 00000020:00010000:2.0:1789760795.969143:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.969147:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.969151:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.969154:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.969156:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.969164:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.969171:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.969190:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c900 00000008:00100000:2.0:1789760795.969312:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27800 - object 0x280000401:2 offset 81661952<>81666048 00000100:00100000:2.0:1789760795.969354:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760795.969372:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503942144:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760795.985021:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987244 gen 2 00000008:00010000:3.0:1789760795.985108:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760795.985116:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.985128:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.985148:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760795.985179:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503942144:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760795.985471:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19937, 19937] 00000080:00200000:2.0:1789760795.985489:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.985495:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.985511:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81666048 i_blocks 159528 00000020:00200000:2.0:1789760795.985514:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.985523:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.985565:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81666048, count: 4096 00000080:00200000:2.0:1789760795.985573:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81666048, bytes: 4096 00000080:00200000:2.0:1789760795.985582:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.985592:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760795.985594:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81666048, 81670144) 0 0 00000080:00200000:2.0:1789760795.985599:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.985615:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19938, 19938] 00020000:00200000:2.0:1789760795.985624:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81666048 chunk: [81666048, 81670144) 81670144, 4096 00020000:00200000:2.0:1789760795.985628:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.985648:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19938, 19938] 00000008:00010000:2.0:1789760795.985665:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.985671:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760795.985682:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81666048, 81670144) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.985696:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81666048=0x4de2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.985758:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.985760:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81670144 00000020:00010000:2.0:1789760795.985770:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.985773:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.985776:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.985780:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.985782:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.985791:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.985798:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.985814:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000008:00100000:2.0:1789760795.985917:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25180 - object 0x280000401:2 offset 81666048<>81670144 00000100:00100000:3.0:1789760795.986239:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503942272:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.014264:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987245 gen 2 00000008:00010000:3.0:1789760796.014319:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760796.014328:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.014340:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.014493:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760796.014529:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503942272:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.014984:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19938, 19938] 00000080:00200000:2.0:1789760796.015003:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.015009:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.015026:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81670144 i_blocks 159536 00000020:00200000:2.0:1789760796.015030:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.015062:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.015099:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81670144, count: 4096 00000080:00200000:2.0:1789760796.015107:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81670144, bytes: 4096 00000080:00200000:2.0:1789760796.015119:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.015128:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.015131:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81670144, 81674240) 0 0 00000080:00200000:2.0:1789760796.015136:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.015153:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19939, 19939] 00020000:00200000:2.0:1789760796.015163:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81670144 chunk: [81670144, 81674240) 81674240, 4096 00020000:00200000:2.0:1789760796.015168:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.015190:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19939, 19939] 00000008:00010000:2.0:1789760796.015210:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.015219:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.015230:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81670144, 81674240) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.015249:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81670144=0x4de3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.017187:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.017198:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81674240 00000020:00010000:2.0:1789760796.017213:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.017218:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.017222:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.017225:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.017228:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.017243:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.017253:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.017287:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cb40 00000008:00100000:2.0:1789760796.017420:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25c00 - object 0x280000401:2 offset 81670144<>81674240 00000100:00100000:3.0:1789760796.017729:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503942400:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.036026:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987246 gen 2 00000008:00010000:3.0:1789760796.036172:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760796.036182:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.036196:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.036222:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760796.036259:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503942400:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.036512:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19939, 19939] 00000080:00200000:2.0:1789760796.036536:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.036543:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.036571:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81674240 i_blocks 159544 00000020:00200000:2.0:1789760796.036575:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.036586:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.036629:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81674240, count: 4096 00000080:00200000:2.0:1789760796.036640:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81674240, bytes: 4096 00000080:00200000:2.0:1789760796.036656:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.036667:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.036669:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81674240, 81678336) 0 0 00000080:00200000:2.0:1789760796.036675:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.036788:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19940, 19940] 00020000:00200000:2.0:1789760796.036806:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81674240 chunk: [81674240, 81678336) 81678336, 4096 00020000:00200000:2.0:1789760796.036815:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.036841:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19940, 19940] 00000008:00010000:2.0:1789760796.036862:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.036867:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.036880:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81674240, 81678336) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.036893:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81674240=0x4de4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.036959:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.036963:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81678336 00000020:00010000:2.0:1789760796.036970:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.036973:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.036976:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.036980:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.036982:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.037004:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.037012:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.037083:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c600 00000008:00100000:2.0:1789760796.037193:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24380 - object 0x280000401:2 offset 81674240<>81678336 00000100:00100000:3.0:1789760796.037301:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503942528:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.065673:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987247 gen 2 00000008:00010000:3.0:1789760796.065712:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760796.065720:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.065731:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.065750:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760796.065776:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503942528:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.065916:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19940, 19940] 00000080:00200000:2.0:1789760796.065929:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.065933:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.065945:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81678336 i_blocks 159552 00000020:00200000:2.0:1789760796.065948:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.065956:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.065983:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81678336, count: 4096 00000080:00200000:2.0:1789760796.065989:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81678336, bytes: 4096 00000080:00200000:2.0:1789760796.065997:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.066005:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.066007:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81678336, 81682432) 0 0 00000080:00200000:2.0:1789760796.066010:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.066022:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19941, 19941] 00020000:00200000:2.0:1789760796.066029:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81678336 chunk: [81678336, 81682432) 81682432, 4096 00020000:00200000:2.0:1789760796.066057:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.066073:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19941, 19941] 00000008:00010000:2.0:1789760796.066088:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.066093:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.066101:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81678336, 81682432) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.066112:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81678336=0x4de5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.066155:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.066158:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81682432 00000020:00010000:2.0:1789760796.066165:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.066168:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.066171:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.066174:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.066176:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.066183:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.066189:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.066203:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c540 00000008:00100000:2.0:1789760796.066291:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24700 - object 0x280000401:2 offset 81678336<>81682432 00000100:00100000:3.0:1789760796.066535:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503942656:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.085009:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987248 gen 2 00000008:00010000:3.0:1789760796.085085:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760796.085092:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.085100:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.085119:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760796.085143:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503942656:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.085231:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19941, 19941] 00000080:00200000:2.0:1789760796.085245:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.085251:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.085263:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81682432 i_blocks 159560 00000020:00200000:2.0:1789760796.085266:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.085274:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.085299:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81682432, count: 4096 00000080:00200000:2.0:1789760796.085306:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81682432, bytes: 4096 00000080:00200000:2.0:1789760796.085314:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.085321:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.085323:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81682432, 81686528) 0 0 00000080:00200000:2.0:1789760796.085327:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.085339:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19942, 19942] 00020000:00200000:2.0:1789760796.085347:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81682432 chunk: [81682432, 81686528) 81686528, 4096 00020000:00200000:2.0:1789760796.085350:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.085365:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19942, 19942] 00000008:00010000:2.0:1789760796.085379:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.085384:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.085393:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81682432, 81686528) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.085404:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81682432=0x4de6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.085452:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.085455:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81686528 00000020:00010000:2.0:1789760796.085463:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.085466:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.085469:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.085473:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.085476:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.085483:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.085489:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.085513:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000008:00100000:2.0:1789760796.085840:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 81682432<>81686528 00000100:00100000:3.0:1789760796.086274:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503942784:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.105895:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987249 gen 2 00000008:00010000:3.0:1789760796.105935:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760796.105942:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.105949:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.105969:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760796.105992:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503942784:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.107726:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19942, 19942] 00000080:00200000:2.0:1789760796.107743:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.107748:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.107761:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81686528 i_blocks 159568 00000020:00200000:2.0:1789760796.107766:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.107777:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.107804:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81686528, count: 4096 00000080:00200000:2.0:1789760796.107810:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81686528, bytes: 4096 00000080:00200000:2.0:1789760796.107820:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.107827:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.107829:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81686528, 81690624) 0 0 00000080:00200000:2.0:1789760796.107832:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.107844:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19943, 19943] 00020000:00200000:2.0:1789760796.107850:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81686528 chunk: [81686528, 81690624) 81690624, 4096 00020000:00200000:2.0:1789760796.107854:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.107868:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19943, 19943] 00000008:00010000:2.0:1789760796.107880:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.107884:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.107893:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81686528, 81690624) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.107904:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81686528=0x4de7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.107950:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.107953:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81690624 00000020:00010000:2.0:1789760796.107958:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.107961:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.107964:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.107967:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.107970:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.107977:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.107982:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.107997:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c840 00000008:00100000:2.0:1789760796.108124:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24e00 - object 0x280000401:2 offset 81686528<>81690624 00000100:00100000:3.0:1789760796.109069:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503942912:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.122795:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987250 gen 2 00000008:00010000:3.0:1789760796.122831:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760796.122838:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.122847:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.122864:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760796.122890:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503942912:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.123220:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19943, 19943] 00000080:00200000:2.0:1789760796.123233:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.123237:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.123249:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81690624 i_blocks 159576 00000020:00200000:2.0:1789760796.123252:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.123259:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.123293:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81690624, count: 4096 00000080:00200000:2.0:1789760796.123299:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81690624, bytes: 4096 00000080:00200000:2.0:1789760796.123306:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.123312:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.123315:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81690624, 81694720) 0 0 00000080:00200000:2.0:1789760796.123318:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.123329:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19944, 19944] 00020000:00200000:2.0:1789760796.123335:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81690624 chunk: [81690624, 81694720) 81694720, 4096 00020000:00200000:2.0:1789760796.123339:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.123353:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19944, 19944] 00000008:00010000:2.0:1789760796.123365:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.123369:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.123377:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81690624, 81694720) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.123387:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81690624=0x4de8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.123427:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.123429:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81694720 00000020:00010000:2.0:1789760796.123435:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.123438:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.123441:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.123444:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.123446:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.123453:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.123458:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.123470:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cd80 00000008:00100000:2.0:1789760796.123565:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27480 - object 0x280000401:2 offset 81690624<>81694720 00000100:00100000:3.0:1789760796.123823:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503943040:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.141025:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987251 gen 2 00000008:00010000:3.0:1789760796.141099:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760796.141105:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.141114:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.141133:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760796.141156:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503943040:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.141278:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19944, 19944] 00000080:00200000:2.0:1789760796.141295:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.141300:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.141311:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81694720 i_blocks 159584 00000020:00200000:2.0:1789760796.141315:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.141322:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.141345:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81694720, count: 4096 00000080:00200000:2.0:1789760796.141351:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81694720, bytes: 4096 00000080:00200000:2.0:1789760796.141357:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.141364:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.141366:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81694720, 81698816) 0 0 00000080:00200000:2.0:1789760796.141370:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.141383:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19945, 19945] 00020000:00200000:2.0:1789760796.141389:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81694720 chunk: [81694720, 81698816) 81698816, 4096 00020000:00200000:2.0:1789760796.141393:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.141407:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19945, 19945] 00000008:00010000:2.0:1789760796.141421:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.141425:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.141434:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81694720, 81698816) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.141444:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81694720=0x4de9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.141487:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.141489:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81698816 00000020:00010000:2.0:1789760796.141496:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.141499:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.141502:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.141505:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.141508:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.141514:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.141521:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.141533:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c180 00000008:00100000:2.0:1789760796.141631:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26680 - object 0x280000401:2 offset 81694720<>81698816 00000100:00100000:3.0:1789760796.142100:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760796.142115:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503943168:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.152797:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987252 gen 2 00000008:00010000:3.0:1789760796.152833:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760796.152839:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.152847:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.152865:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760796.152889:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503943168:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.152968:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19945, 19945] 00000080:00200000:2.0:1789760796.152980:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.152986:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.152998:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81698816 i_blocks 159592 00000020:00200000:2.0:1789760796.153001:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.153008:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.153056:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81698816, count: 4096 00000080:00200000:2.0:1789760796.153063:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81698816, bytes: 4096 00000080:00200000:2.0:1789760796.153069:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.153076:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.153078:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81698816, 81702912) 0 0 00000080:00200000:2.0:1789760796.153082:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.153094:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19946, 19946] 00020000:00200000:2.0:1789760796.153101:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81698816 chunk: [81698816, 81702912) 81702912, 4096 00020000:00200000:2.0:1789760796.153104:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.153118:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19946, 19946] 00000008:00010000:2.0:1789760796.153131:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.153135:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.153143:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81698816, 81702912) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.153155:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81698816=0x4dea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.153194:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.153197:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81702912 00000020:00010000:2.0:1789760796.153203:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.153206:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.153209:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.153212:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.153214:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.153221:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.153227:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.153240:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c780 00000008:00100000:2.0:1789760796.153328:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25500 - object 0x280000401:2 offset 81698816<>81702912 00000100:00100000:3.0:1789760796.153465:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760796.153480:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503943296:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.167090:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987253 gen 2 00000008:00010000:3.0:1789760796.167122:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760796.167129:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.167137:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.167155:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760796.167177:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503943296:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.167262:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19946, 19946] 00000080:00200000:2.0:1789760796.167274:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.167279:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.167300:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81702912 i_blocks 159600 00000020:00200000:2.0:1789760796.167303:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.167310:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.167334:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81702912, count: 4096 00000080:00200000:2.0:1789760796.167340:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81702912, bytes: 4096 00000080:00200000:2.0:1789760796.167347:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.167353:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.167355:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81702912, 81707008) 0 0 00000080:00200000:2.0:1789760796.167359:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.167372:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19947, 19947] 00020000:00200000:2.0:1789760796.167378:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81702912 chunk: [81702912, 81707008) 81707008, 4096 00020000:00200000:2.0:1789760796.167381:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.167395:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19947, 19947] 00000008:00010000:2.0:1789760796.167409:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.167414:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.167423:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81702912, 81707008) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.167434:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81702912=0x4deb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.167481:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.167484:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81707008 00000020:00010000:2.0:1789760796.167500:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.167504:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.167506:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.167509:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.167512:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.167519:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.167525:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.167539:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c780 00000008:00100000:2.0:1789760796.167979:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24000 - object 0x280000401:2 offset 81702912<>81707008 00000100:00100000:3.0:1789760796.168104:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760796.168135:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503943424:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.179028:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987254 gen 2 00000008:00010000:3.0:1789760796.179133:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760796.179143:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.179155:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.179177:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760796.179213:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503943424:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.179811:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19947, 19947] 00000080:00200000:2.0:1789760796.179829:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.179836:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.179852:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81707008 i_blocks 159608 00000020:00200000:2.0:1789760796.179857:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.179868:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.179904:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81707008, count: 4096 00000080:00200000:2.0:1789760796.179913:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81707008, bytes: 4096 00000080:00200000:2.0:1789760796.179927:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.179937:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.179939:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81707008, 81711104) 0 0 00000080:00200000:2.0:1789760796.179943:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.179960:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19948, 19948] 00020000:00200000:2.0:1789760796.179969:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81707008 chunk: [81707008, 81711104) 81711104, 4096 00020000:00200000:2.0:1789760796.179974:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.179996:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19948, 19948] 00000008:00010000:2.0:1789760796.180016:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.180023:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.180060:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81707008, 81711104) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.180074:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81707008=0x4dec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.180135:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.180138:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81711104 00000020:00010000:2.0:1789760796.180146:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.180149:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.180152:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.180156:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.180159:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.180169:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.180175:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.180194:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c180 00000008:00100000:2.0:1789760796.180299:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24000 - object 0x280000401:2 offset 81707008<>81711104 00000100:00100000:3.0:1789760796.180791:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760796.180821:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503943552:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.198133:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987255 gen 2 00000008:00010000:3.0:1789760796.198183:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760796.198193:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.198204:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.198264:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760796.198296:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503943552:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.198853:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19948, 19948] 00000080:00200000:2.0:1789760796.198872:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.198879:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.198897:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81711104 i_blocks 159616 00000020:00200000:2.0:1789760796.198901:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.198912:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.198946:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81711104, count: 4096 00000080:00200000:2.0:1789760796.198954:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81711104, bytes: 4096 00000080:00200000:2.0:1789760796.198965:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.198975:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.198977:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81711104, 81715200) 0 0 00000080:00200000:2.0:1789760796.198983:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.198998:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19949, 19949] 00020000:00200000:2.0:1789760796.199007:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81711104 chunk: [81711104, 81715200) 81715200, 4096 00020000:00200000:2.0:1789760796.199012:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.199029:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19949, 19949] 00000008:00010000:2.0:1789760796.199069:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.199075:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.199087:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81711104, 81715200) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.199101:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81711104=0x4ded000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.199159:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.199162:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81715200 00000020:00010000:2.0:1789760796.199170:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.199174:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.199177:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.199181:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.199184:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.199193:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.199200:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.199216:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cd80 00000008:00100000:2.0:1789760796.199315:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25500 - object 0x280000401:2 offset 81711104<>81715200 00000100:00100000:3.0:1789760796.199384:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503943680:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.215298:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987256 gen 2 00000008:00010000:3.0:1789760796.215350:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760796.215358:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.215368:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.215393:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760796.215423:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503943680:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760796.215485:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19949, 19949] 00000080:00200000:0.0:1789760796.215503:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.215509:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.215524:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81715200 i_blocks 159624 00000020:00200000:0.0:1789760796.215528:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.215538:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.215597:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81715200, count: 4096 00000080:00200000:0.0:1789760796.215605:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81715200, bytes: 4096 00000080:00200000:0.0:1789760796.215614:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.215624:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760796.215626:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81715200, 81719296) 0 0 00000080:00200000:0.0:1789760796.215630:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.215647:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19950, 19950] 00020000:00200000:0.0:1789760796.215656:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81715200 chunk: [81715200, 81719296) 81719296, 4096 00020000:00200000:0.0:1789760796.215661:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.215679:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19950, 19950] 00000008:00010000:0.0:1789760796.215696:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.215702:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760796.215713:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81715200, 81719296) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.215794:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81715200=0x4dee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.215856:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.215858:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81719296 00000020:00010000:0.0:1789760796.215867:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.215870:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.215873:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.215877:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.215881:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.215889:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.215896:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.215914:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760796.217187:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 81715200<>81719296 00000100:00100000:0.0:1789760796.217263:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503943808:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.228951:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987257 gen 2 00000008:00010000:0.0:1789760796.229008:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:0.0:1789760796.229016:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.229027:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.230217:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000080:00200000:1.0:1789760796.230245:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19950, 19950] 00000100:00100000:0.0:1789760796.230265:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503943808:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.230265:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.230271:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.230296:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81719296 i_blocks 159632 00000020:00200000:1.0:1789760796.230302:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.230314:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.230352:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81719296, count: 4096 00000080:00200000:1.0:1789760796.230361:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81719296, bytes: 4096 00000080:00200000:1.0:1789760796.230375:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.230384:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.230386:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81719296, 81723392) 0 0 00000080:00200000:1.0:1789760796.230390:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.230407:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19951, 19951] 00020000:00200000:1.0:1789760796.230417:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81719296 chunk: [81719296, 81723392) 81723392, 4096 00020000:00200000:1.0:1789760796.230421:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.230447:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19951, 19951] 00000008:00010000:1.0:1789760796.230467:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.230474:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.230486:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81719296, 81723392) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.230500:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81719296=0x4def000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.230633:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.230636:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81723392 00000020:00010000:1.0:1789760796.230645:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.230648:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.230652:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.230655:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.230658:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.230667:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.230673:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.230690:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760796.230857:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81719296<>81723392 00000100:00100000:0.0:1789760796.231028:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503943936:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.248272:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987258 gen 2 00000008:00010000:0.0:1789760796.248340:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760796.248349:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.248361:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.248387:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760796.248419:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503943936:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.248763:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19951, 19951] 00000080:00200000:1.0:1789760796.248784:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.248790:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.248808:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81723392 i_blocks 159640 00000020:00200000:1.0:1789760796.248813:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.248823:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.248862:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81723392, count: 4096 00000080:00200000:1.0:1789760796.248870:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81723392, bytes: 4096 00000080:00200000:1.0:1789760796.248881:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.248891:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.248894:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81723392, 81727488) 0 0 00000080:00200000:1.0:1789760796.248899:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.248917:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19952, 19952] 00020000:00200000:1.0:1789760796.248927:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81723392 chunk: [81723392, 81727488) 81727488, 4096 00020000:00200000:1.0:1789760796.248932:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.248954:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19952, 19952] 00000008:00010000:1.0:1789760796.248975:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.248982:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.248994:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81723392, 81727488) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.249010:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81723392=0x4df0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.249092:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.249095:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81727488 00000020:00010000:1.0:1789760796.249104:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.249107:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.249110:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.249114:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.249117:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.249137:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.249145:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.249163:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760796.249270:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81723392<>81727488 00000100:00100000:0.0:1789760796.249533:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503944064:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.266903:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987259 gen 2 00000008:00010000:0.0:1789760796.266957:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760796.266966:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.266977:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.267675:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:0.0:1789760796.267707:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503944064:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.268439:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19952, 19952] 00000080:00200000:1.0:1789760796.268465:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.268472:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.268487:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81727488 i_blocks 159648 00000020:00200000:1.0:1789760796.268497:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.268508:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.268542:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81727488, count: 4096 00000080:00200000:1.0:1789760796.268549:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81727488, bytes: 4096 00000080:00200000:1.0:1789760796.268569:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.268579:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.268581:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81727488, 81731584) 0 0 00000080:00200000:1.0:1789760796.268586:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.268611:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19953, 19953] 00020000:00200000:1.0:1789760796.268620:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81727488 chunk: [81727488, 81731584) 81731584, 4096 00020000:00200000:1.0:1789760796.268625:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.268766:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19953, 19953] 00000008:00010000:1.0:1789760796.268787:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.268792:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.268803:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81727488, 81731584) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.268817:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81727488=0x4df1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.268874:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.268877:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81731584 00000020:00010000:1.0:1789760796.268884:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.268888:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.268891:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.268895:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.268898:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.268905:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.268912:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.268933:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760796.269078:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 81727488<>81731584 00000100:00100000:0.0:1789760796.269474:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503944192:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.285744:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987260 gen 2 00000008:00010000:0.0:1789760796.285797:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760796.285807:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.285819:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.285844:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000100:00100000:0.0:1789760796.285876:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503944192:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.286009:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19953, 19953] 00000080:00200000:1.0:1789760796.286027:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.286068:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.286085:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81731584 i_blocks 159656 00000020:00200000:1.0:1789760796.286088:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.286098:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.286149:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81731584, count: 4096 00000080:00200000:1.0:1789760796.286157:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81731584, bytes: 4096 00000080:00200000:1.0:1789760796.286170:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.286179:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.286181:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81731584, 81735680) 0 0 00000080:00200000:1.0:1789760796.286186:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.286203:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19954, 19954] 00020000:00200000:1.0:1789760796.286211:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81731584 chunk: [81731584, 81735680) 81735680, 4096 00020000:00200000:1.0:1789760796.286216:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.286235:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19954, 19954] 00000008:00010000:1.0:1789760796.286281:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.286296:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.286308:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81731584, 81735680) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.286323:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81731584=0x4df2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.286383:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.286387:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81735680 00000020:00010000:1.0:1789760796.286395:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.286400:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.286402:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.286406:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.286409:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.286417:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.286424:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.286440:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760796.286564:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 81731584<>81735680 00000100:00100000:0.0:1789760796.286782:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503944320:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.299862:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987261 gen 2 00000008:00010000:0.0:1789760796.299914:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760796.299922:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.299934:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.299957:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000100:00100000:0.0:1789760796.299991:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503944320:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.300202:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19954, 19954] 00000080:00200000:1.0:1789760796.300217:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.300223:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.300239:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81735680 i_blocks 159664 00000020:00200000:1.0:1789760796.300242:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.300253:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.300435:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81735680, count: 4096 00000080:00200000:1.0:1789760796.300444:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81735680, bytes: 4096 00000080:00200000:1.0:1789760796.300456:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.300465:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.300467:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81735680, 81739776) 0 0 00000080:00200000:1.0:1789760796.300472:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.300488:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19955, 19955] 00020000:00200000:1.0:1789760796.300496:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81735680 chunk: [81735680, 81739776) 81739776, 4096 00020000:00200000:1.0:1789760796.300501:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.300520:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19955, 19955] 00000008:00010000:1.0:1789760796.300537:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.300543:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.300563:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81735680, 81739776) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.300579:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81735680=0x4df3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.300787:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.300791:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81739776 00000020:00010000:1.0:1789760796.300800:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.300804:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.300807:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.300811:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.300814:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.300823:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.300830:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.300854:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760796.300966:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 81735680<>81739776 00000100:00100000:0.0:1789760796.301404:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503944448:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.317802:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987262 gen 2 00000008:00010000:0.0:1789760796.317855:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760796.317864:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.317889:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.317913:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:0.0:1789760796.317945:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503944448:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.318777:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19955, 19955] 00000080:00200000:1.0:1789760796.318798:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.318805:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.318821:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81739776 i_blocks 159672 00000020:00200000:1.0:1789760796.318825:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.318836:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.318882:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81739776, count: 4096 00000080:00200000:1.0:1789760796.318890:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81739776, bytes: 4096 00000080:00200000:1.0:1789760796.318902:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.318911:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.318914:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81739776, 81743872) 0 0 00000080:00200000:1.0:1789760796.318918:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.318934:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19956, 19956] 00020000:00200000:1.0:1789760796.318943:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81739776 chunk: [81739776, 81743872) 81743872, 4096 00020000:00200000:1.0:1789760796.318947:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.318963:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19956, 19956] 00000008:00010000:1.0:1789760796.318978:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.318983:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.318993:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81739776, 81743872) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.319005:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81739776=0x4df4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.319088:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.319092:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81743872 00000020:00010000:1.0:1789760796.319100:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.319103:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.319107:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.319110:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.319113:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.319120:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.319135:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.319151:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760796.319253:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 81739776<>81743872 00000100:00100000:0.0:1789760796.319519:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503944576:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.336124:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987263 gen 2 00000008:00010000:0.0:1789760796.336174:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760796.336182:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.336195:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.336221:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760796.336251:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503944576:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.336375:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19956, 19956] 00000080:00200000:1.0:1789760796.336393:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.336400:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.336415:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81743872 i_blocks 159680 00000020:00200000:1.0:1789760796.336419:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.336428:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.336463:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81743872, count: 4096 00000080:00200000:1.0:1789760796.336471:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81743872, bytes: 4096 00000080:00200000:1.0:1789760796.336480:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.336500:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.336503:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81743872, 81747968) 0 0 00000080:00200000:1.0:1789760796.336507:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.336524:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19957, 19957] 00020000:00200000:1.0:1789760796.336533:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81743872 chunk: [81743872, 81747968) 81747968, 4096 00020000:00200000:1.0:1789760796.336538:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.336569:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19957, 19957] 00000008:00010000:1.0:1789760796.336589:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.336605:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.336616:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81743872, 81747968) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.336630:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81743872=0x4df5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.336683:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.336687:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81747968 00000020:00010000:1.0:1789760796.336694:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.336698:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.336701:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.336705:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.336708:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.336716:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.336723:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.336740:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760796.336843:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 81743872<>81747968 00000100:00100000:0.0:1789760796.337142:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503944704:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.350004:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987264 gen 2 00000008:00010000:0.0:1789760796.350093:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:0.0:1789760796.350102:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.350113:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.350126:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:0.0:1789760796.350158:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503944704:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.350275:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19957, 19957] 00000080:00200000:1.0:1789760796.350292:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.350298:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.350313:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81747968 i_blocks 159688 00000020:00200000:1.0:1789760796.350317:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.350327:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.350362:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81747968, count: 4096 00000080:00200000:1.0:1789760796.350369:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81747968, bytes: 4096 00000080:00200000:1.0:1789760796.350382:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.350390:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.350393:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81747968, 81752064) 0 0 00000080:00200000:1.0:1789760796.350397:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.350411:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19958, 19958] 00020000:00200000:1.0:1789760796.350418:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81747968 chunk: [81747968, 81752064) 81752064, 4096 00020000:00200000:1.0:1789760796.350421:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.350437:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19958, 19958] 00000008:00010000:1.0:1789760796.350451:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.350456:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.350465:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81747968, 81752064) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.350476:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81747968=0x4df6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.350530:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.350534:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81752064 00000020:00010000:1.0:1789760796.350543:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.350546:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.350549:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.350672:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.350677:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.350686:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.350693:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.350711:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58780 00000008:00100000:1.0:1789760796.350976:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 81747968<>81752064 00000100:00100000:0.0:1789760796.351105:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503944832:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.371863:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987265 gen 2 00000008:00010000:0.0:1789760796.371928:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58780 00000020:00200000:0.0:1789760796.371936:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.371956:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.371971:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58780 00000100:00100000:0.0:1789760796.372004:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503944832:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.372436:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19958, 19958] 00000080:00200000:1.0:1789760796.372454:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.372460:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.372476:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81752064 i_blocks 159696 00000020:00200000:1.0:1789760796.372480:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.373794:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.373843:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81752064, count: 4096 00000080:00200000:1.0:1789760796.373855:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81752064, bytes: 4096 00000080:00200000:1.0:1789760796.373939:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.374013:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.374016:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81752064, 81756160) 0 0 00000080:00200000:1.0:1789760796.374023:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.374068:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19959, 19959] 00020000:00200000:1.0:1789760796.374077:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81752064 chunk: [81752064, 81756160) 81756160, 4096 00020000:00200000:1.0:1789760796.374082:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.374104:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19959, 19959] 00000008:00010000:1.0:1789760796.374123:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.374129:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.374140:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81752064, 81756160) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.374154:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81752064=0x4df7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.374220:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.374225:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81756160 00000020:00010000:1.0:1789760796.374233:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.374237:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.374241:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.374245:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.374248:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.374259:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.374266:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.374516:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760796.374656:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 81752064<>81756160 00000100:00100000:0.0:1789760796.376205:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760796.376243:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503944960:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.389000:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987266 gen 2 00000008:00010000:0.0:1789760796.389090:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:0.0:1789760796.389099:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.389110:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.389177:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000100:00100000:0.0:1789760796.389210:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503944960:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760796.389282:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19959, 19959] 00000080:00200000:3.0:1789760796.389358:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.389364:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.389379:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81756160 i_blocks 159704 00000020:00200000:3.0:1789760796.389382:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.389390:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.389421:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81756160, count: 4096 00000080:00200000:3.0:1789760796.389427:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81756160, bytes: 4096 00000080:00200000:3.0:1789760796.389436:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.389443:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760796.389445:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81756160, 81760256) 0 0 00000080:00200000:3.0:1789760796.389448:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.389462:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19960, 19960] 00020000:00200000:3.0:1789760796.389469:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81756160 chunk: [81756160, 81760256) 81760256, 4096 00020000:00200000:3.0:1789760796.389475:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.389497:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19960, 19960] 00000008:00010000:3.0:1789760796.389516:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.389522:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760796.389533:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81756160, 81760256) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.389547:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81756160=0x4df8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.389614:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.389617:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81760256 00000020:00010000:3.0:1789760796.389625:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.389628:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.389631:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.389699:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.389704:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.389712:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.389720:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.389796:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760796.389969:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070fb80 - object 0x280000401:2 offset 81756160<>81760256 00000100:00100000:3.0:1789760796.390022:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760796.390068:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503945088:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.402814:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987267 gen 2 00000008:00010000:3.0:1789760796.402868:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760796.402887:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.402899:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.402921:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760796.402951:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503945088:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.403116:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19960, 19960] 00000080:00200000:2.0:1789760796.403133:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.403139:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.403156:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81760256 i_blocks 159712 00000020:00200000:2.0:1789760796.403159:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.403170:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.403208:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81760256, count: 4096 00000080:00200000:2.0:1789760796.403216:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81760256, bytes: 4096 00000080:00200000:2.0:1789760796.403226:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.403235:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.403237:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81760256, 81764352) 0 0 00000080:00200000:2.0:1789760796.403242:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.403258:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19961, 19961] 00020000:00200000:2.0:1789760796.403269:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81760256 chunk: [81760256, 81764352) 81764352, 4096 00020000:00200000:2.0:1789760796.403273:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.403308:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19961, 19961] 00000008:00010000:2.0:1789760796.403327:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.403333:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.403345:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81760256, 81764352) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.403359:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81760256=0x4df9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.403419:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.403422:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81764352 00000020:00010000:2.0:1789760796.403429:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.403432:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.403435:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.403438:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.403441:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.403448:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.403456:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.403471:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c840 00000008:00100000:2.0:1789760796.404000:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26680 - object 0x280000401:2 offset 81760256<>81764352 00000100:00100000:2.0:1789760796.404192:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760796.404212:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503945216:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.413513:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987268 gen 2 00000008:00010000:3.0:1789760796.413583:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760796.413603:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.413615:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.413767:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19961, 19961] 00000080:00200000:2.0:1789760796.413785:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.413791:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.413807:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81764352 i_blocks 159720 00000020:00200000:2.0:1789760796.413811:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.413821:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.413856:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81764352, count: 4096 00000008:00010000:3.0:1789760796.413885:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760796.413916:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503945216:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.414022:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81764352, bytes: 4096 00000080:00200000:2.0:1789760796.414232:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.414242:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.414244:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81764352, 81768448) 0 0 00000080:00200000:2.0:1789760796.414248:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.414266:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19962, 19962] 00020000:00200000:2.0:1789760796.414276:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81764352 chunk: [81764352, 81768448) 81768448, 4096 00020000:00200000:2.0:1789760796.414291:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.414314:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19962, 19962] 00000008:00010000:2.0:1789760796.414334:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.414340:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.414351:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81764352, 81768448) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.414365:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81764352=0x4dfa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.414426:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.414429:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81768448 00000020:00010000:2.0:1789760796.414438:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.414442:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.414444:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.414448:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.414450:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.414460:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.414466:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.414484:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000008:00100000:2.0:1789760796.414911:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27480 - object 0x280000401:2 offset 81764352<>81768448 00000100:00100000:3.0:1789760796.415195:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503945344:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.438411:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987269 gen 2 00000008:00010000:2.0:1789760796.439413:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:2.0:1789760796.439426:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.439437:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.439542:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000080:00200000:0.0:1789760796.440031:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19962, 19962] 00000100:00100000:2.0:1789760796.440087:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503945344:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760796.440200:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.440206:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.440224:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81768448 i_blocks 159728 00000020:00200000:0.0:1789760796.440242:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.440254:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.440293:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81768448, count: 4096 00000080:00200000:0.0:1789760796.440306:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81768448, bytes: 4096 00000080:00200000:0.0:1789760796.440324:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.440334:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760796.440337:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81768448, 81772544) 0 0 00000080:00200000:0.0:1789760796.440342:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.440360:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19963, 19963] 00020000:00200000:0.0:1789760796.440370:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81768448 chunk: [81768448, 81772544) 81772544, 4096 00020000:00200000:0.0:1789760796.440375:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.440398:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19963, 19963] 00000008:00010000:0.0:1789760796.440418:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.440424:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760796.440436:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81768448, 81772544) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.440451:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81768448=0x4dfb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.440515:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.440518:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81772544 00000020:00010000:0.0:1789760796.440527:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.440530:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.440534:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.440537:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.440540:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.440550:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.440570:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.440592:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760796.440779:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 81768448<>81772544 00000100:00100000:1.0:1789760796.440863:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503945472:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760796.454630:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987270 gen 2 00000008:00010000:1.0:1789760796.454691:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760796.454699:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.454711:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.454785:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760796.454818:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503945472:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760796.455424:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19963, 19963] 00000080:00200000:0.0:1789760796.455443:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.455449:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.455466:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81772544 i_blocks 159736 00000020:00200000:0.0:1789760796.455471:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.455481:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.455917:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81772544, count: 4096 00000080:00200000:0.0:1789760796.455927:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81772544, bytes: 4096 00000080:00200000:0.0:1789760796.455941:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.455950:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760796.455952:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81772544, 81776640) 0 0 00000080:00200000:0.0:1789760796.455957:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.455974:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19964, 19964] 00020000:00200000:0.0:1789760796.455983:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81772544 chunk: [81772544, 81776640) 81776640, 4096 00020000:00200000:0.0:1789760796.455988:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.456011:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19964, 19964] 00000008:00010000:0.0:1789760796.456030:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.456740:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760796.456759:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81772544, 81776640) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.456777:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81772544=0x4dfc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.456857:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.456860:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81776640 00000020:00010000:0.0:1789760796.456870:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.456874:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.456877:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.456881:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.456884:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.456896:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.456904:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.456934:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760796.458091:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99880 - object 0x280000401:2 offset 81772544<>81776640 00000100:00100000:1.0:1789760796.460247:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503945600:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760796.483505:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987271 gen 2 00000008:00010000:1.0:1789760796.483575:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760796.483584:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.483595:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.483618:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760796.483647:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503945600:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760796.484086:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19964, 19964] 00000080:00200000:0.0:1789760796.484112:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.484119:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.484136:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81776640 i_blocks 159744 00000020:00200000:0.0:1789760796.484140:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.484150:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.484186:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81776640, count: 4096 00000080:00200000:0.0:1789760796.484194:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81776640, bytes: 4096 00000080:00200000:0.0:1789760796.484204:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.484214:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760796.484216:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81776640, 81780736) 0 0 00000080:00200000:0.0:1789760796.484221:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.484237:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19965, 19965] 00020000:00200000:0.0:1789760796.484246:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81776640 chunk: [81776640, 81780736) 81780736, 4096 00020000:00200000:0.0:1789760796.484251:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.484271:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19965, 19965] 00000008:00010000:0.0:1789760796.484289:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.484295:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760796.484307:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81776640, 81780736) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.484322:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81776640=0x4dfd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.484386:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.484389:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81780736 00000020:00010000:0.0:1789760796.484398:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.484401:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.484405:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.484408:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.484411:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.484420:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.484441:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.484459:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760796.486745:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99f80 - object 0x280000401:2 offset 81776640<>81780736 00000100:00100000:1.0:1789760796.486844:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760796.486877:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503945728:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760796.500451:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987272 gen 2 00000008:00010000:1.0:1789760796.500507:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760796.500516:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.500529:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.500570:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760796.500618:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503945728:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760796.500775:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19965, 19965] 00000080:00200000:0.0:1789760796.500791:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.500797:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.500813:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81780736 i_blocks 159752 00000020:00200000:0.0:1789760796.500817:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.500828:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.500861:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81780736, count: 4096 00000080:00200000:0.0:1789760796.500869:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81780736, bytes: 4096 00000080:00200000:0.0:1789760796.500882:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.500890:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760796.500893:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81780736, 81784832) 0 0 00000080:00200000:0.0:1789760796.500897:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.500915:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19966, 19966] 00020000:00200000:0.0:1789760796.500924:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81780736 chunk: [81780736, 81784832) 81784832, 4096 00020000:00200000:0.0:1789760796.500928:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.500948:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19966, 19966] 00000008:00010000:0.0:1789760796.500966:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.500972:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760796.500984:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81780736, 81784832) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.500998:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81780736=0x4dfe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.501426:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.501430:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81784832 00000020:00010000:0.0:1789760796.501439:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.501443:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.501446:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.501450:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.501453:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.501461:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.501469:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.501548:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:1.0:1789760796.501912:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81780736<>81784832 00000100:00100000:1.0:1789760796.501978:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503945856:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760796.524220:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987273 gen 2 00000008:00010000:1.0:1789760796.524272:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760796.524280:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.524292:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.524317:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760796.524346:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503945856:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760796.524452:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19966, 19966] 00000080:00200000:0.0:1789760796.524469:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.524475:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.524491:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81784832 i_blocks 159760 00000020:00200000:0.0:1789760796.524495:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.524504:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.524540:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81784832, count: 4096 00000080:00200000:0.0:1789760796.524547:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81784832, bytes: 4096 00000080:00200000:0.0:1789760796.524568:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.524577:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760796.524579:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81784832, 81788928) 0 0 00000080:00200000:0.0:1789760796.524584:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.524600:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19967, 19967] 00020000:00200000:0.0:1789760796.524610:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81784832 chunk: [81784832, 81788928) 81788928, 4096 00020000:00200000:0.0:1789760796.524615:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.524635:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19967, 19967] 00000008:00010000:0.0:1789760796.524653:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.524659:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760796.524671:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81784832, 81788928) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.524685:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81784832=0x4dff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.524743:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.524746:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81788928 00000020:00010000:0.0:1789760796.524754:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.524758:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.524761:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.524765:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.524768:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.524777:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.524783:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.524800:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760796.524904:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9bb80 - object 0x280000401:2 offset 81784832<>81788928 00000100:00100000:1.0:1789760796.524974:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503945984:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760796.549586:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987274 gen 2 00000008:00010000:1.0:1789760796.549649:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760796.549669:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.549682:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.549706:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760796.549739:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503945984:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760796.549915:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19967, 19967] 00000080:00200000:0.0:1789760796.549934:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.549940:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.549955:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81788928 i_blocks 159768 00000020:00200000:0.0:1789760796.549960:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.549970:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.550015:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81788928, count: 4096 00000080:00200000:0.0:1789760796.550022:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81788928, bytes: 4096 00000080:00200000:0.0:1789760796.550074:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.550084:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760796.550087:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81788928, 81793024) 0 0 00000080:00200000:0.0:1789760796.550092:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.550108:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19968, 19968] 00020000:00200000:0.0:1789760796.550116:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81788928 chunk: [81788928, 81793024) 81793024, 4096 00020000:00200000:0.0:1789760796.550121:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.550142:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19968, 19968] 00000008:00010000:0.0:1789760796.550160:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.550167:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760796.550177:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81788928, 81793024) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.550191:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81788928=0x4e00000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.550249:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.550251:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81793024 00000020:00010000:0.0:1789760796.550260:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.550263:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.550267:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.550270:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.550273:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.550281:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.550288:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.550304:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760796.550406:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98700 - object 0x280000401:2 offset 81788928<>81793024 00000100:00100000:1.0:1789760796.550585:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503946112:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760796.568482:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987275 gen 2 00000008:00010000:1.0:1789760796.568548:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760796.568567:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.568581:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.568652:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760796.568693:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503946112:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760796.569761:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19968, 19968] 00000080:00200000:0.0:1789760796.569784:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.569790:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.569809:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81793024 i_blocks 159776 00000020:00200000:0.0:1789760796.569814:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.569828:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.569866:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81793024, count: 4096 00000080:00200000:0.0:1789760796.569874:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81793024, bytes: 4096 00000080:00200000:0.0:1789760796.569889:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.569899:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760796.569901:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81793024, 81797120) 0 0 00000080:00200000:0.0:1789760796.569905:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.569921:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19969, 19969] 00020000:00200000:0.0:1789760796.569930:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81793024 chunk: [81793024, 81797120) 81797120, 4096 00020000:00200000:0.0:1789760796.569935:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.569957:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19969, 19969] 00000008:00010000:0.0:1789760796.569977:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.569983:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760796.569994:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81793024, 81797120) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.570145:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81793024=0x4e01000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.570223:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.570227:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81797120 00000020:00010000:0.0:1789760796.570236:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.570239:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.570243:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.570246:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.570249:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.570258:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.570265:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.570365:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760796.570481:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98700 - object 0x280000401:2 offset 81793024<>81797120 00000100:00100000:1.0:1789760796.570805:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503946240:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760796.585884:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987276 gen 2 00000008:00010000:1.0:1789760796.585937:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760796.585946:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.585959:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.585984:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760796.586031:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503946240:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760796.586780:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19969, 19969] 00000080:00200000:0.0:1789760796.586799:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.586806:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.586825:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81797120 i_blocks 159784 00000020:00200000:0.0:1789760796.586829:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.586841:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.586879:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81797120, count: 4096 00000080:00200000:0.0:1789760796.586886:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81797120, bytes: 4096 00000080:00200000:0.0:1789760796.586900:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.586909:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760796.586911:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81797120, 81801216) 0 0 00000080:00200000:0.0:1789760796.586917:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.586936:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19970, 19970] 00020000:00200000:0.0:1789760796.586944:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81797120 chunk: [81797120, 81801216) 81801216, 4096 00020000:00200000:0.0:1789760796.586950:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.586970:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19970, 19970] 00000008:00010000:0.0:1789760796.586989:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.586996:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760796.587007:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81797120, 81801216) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.587023:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81797120=0x4e02000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.587128:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.587131:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81801216 00000020:00010000:0.0:1789760796.587140:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.587143:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.587146:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.587150:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.587153:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.587161:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.587168:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.587187:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760796.587307:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9bb80 - object 0x280000401:2 offset 81797120<>81801216 00000100:00100000:0.0:1789760796.587365:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503946368:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.601508:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987277 gen 2 00000008:00010000:0.0:1789760796.601563:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:0.0:1789760796.601572:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.601583:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.601701:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:0.0:1789760796.601731:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503946368:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.601847:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19970, 19970] 00000080:00200000:1.0:1789760796.601864:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.601870:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.601887:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81801216 i_blocks 159792 00000020:00200000:1.0:1789760796.601891:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.601903:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.601940:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81801216, count: 4096 00000080:00200000:1.0:1789760796.601948:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81801216, bytes: 4096 00000080:00200000:1.0:1789760796.601959:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.601968:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.601970:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81801216, 81805312) 0 0 00000080:00200000:1.0:1789760796.601974:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.601992:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19971, 19971] 00020000:00200000:1.0:1789760796.602001:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81801216 chunk: [81801216, 81805312) 81805312, 4096 00020000:00200000:1.0:1789760796.602006:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.602027:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19971, 19971] 00000008:00010000:1.0:1789760796.602082:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.602089:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.602101:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81801216, 81805312) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.602115:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81801216=0x4e03000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.602229:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.602232:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81805312 00000020:00010000:1.0:1789760796.602241:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.602244:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.602247:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.602250:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.602253:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.602261:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.602268:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.602345:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760796.602446:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81801216<>81805312 00000100:00100000:0.0:1789760796.602518:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503946496:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.616679:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987278 gen 2 00000008:00010000:0.0:1789760796.616733:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58000 00000020:00200000:0.0:1789760796.616741:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.616753:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.616778:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58000 00000080:00200000:1.0:1789760796.616858:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19971, 19971] 00000080:00200000:1.0:1789760796.616876:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.616882:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.616898:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81805312 i_blocks 159800 00000020:00200000:1.0:1789760796.616902:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.616912:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:0.0:1789760796.616937:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503946496:192.168.203.158@tcp:4:dd.0 00000080:00200008:1.0:1789760796.616948:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81805312, count: 4096 00000080:00200000:1.0:1789760796.616955:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81805312, bytes: 4096 00000080:00200000:1.0:1789760796.616963:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.616973:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.616975:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81805312, 81809408) 0 0 00000080:00200000:1.0:1789760796.616980:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.616997:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19972, 19972] 00020000:00200000:1.0:1789760796.617005:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81805312 chunk: [81805312, 81809408) 81809408, 4096 00020000:00200000:1.0:1789760796.617010:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.617029:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19972, 19972] 00000008:00010000:1.0:1789760796.617089:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.617096:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.617107:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81805312, 81809408) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.617121:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81805312=0x4e04000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.617179:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.617182:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81809408 00000020:00010000:1.0:1789760796.617190:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.617193:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.617196:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.617200:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.617203:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.617209:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.617215:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.617230:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760796.617329:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 81805312<>81809408 00000100:00100000:0.0:1789760796.617727:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503946624:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.632329:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987279 gen 2 00000008:00010000:0.0:1789760796.632381:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:0.0:1789760796.632390:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.632400:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.632426:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:0.0:1789760796.632458:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503946624:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.633412:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19972, 19972] 00000080:00200000:1.0:1789760796.633435:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.633442:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.633462:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81809408 i_blocks 159808 00000020:00200000:1.0:1789760796.633468:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.633481:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.633523:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81809408, count: 4096 00000080:00200000:1.0:1789760796.633531:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81809408, bytes: 4096 00000080:00200000:1.0:1789760796.633547:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.633568:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.633570:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81809408, 81813504) 0 0 00000080:00200000:1.0:1789760796.633576:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.633605:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19973, 19973] 00020000:00200000:1.0:1789760796.633616:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81809408 chunk: [81809408, 81813504) 81813504, 4096 00020000:00200000:1.0:1789760796.633621:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.633642:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19973, 19973] 00000008:00010000:1.0:1789760796.633662:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.633670:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.633681:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81809408, 81813504) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.633696:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81809408=0x4e05000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.633762:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.633765:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81813504 00000020:00010000:1.0:1789760796.633774:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.633778:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.633781:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.633785:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.633787:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.633797:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.633804:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.633824:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760796.633933:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 81809408<>81813504 00000100:00100000:0.0:1789760796.634134:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503946752:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.646215:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987280 gen 2 00000008:00010000:0.0:1789760796.646275:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d586c0 00000020:00200000:0.0:1789760796.646284:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.646296:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.646318:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d586c0 00000100:00100000:0.0:1789760796.646350:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503946752:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.646594:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19973, 19973] 00000080:00200000:1.0:1789760796.646611:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.646617:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.646633:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81813504 i_blocks 159816 00000020:00200000:1.0:1789760796.646637:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.646648:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.646685:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81813504, count: 4096 00000080:00200000:1.0:1789760796.646693:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81813504, bytes: 4096 00000080:00200000:1.0:1789760796.646704:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.646714:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.646716:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81813504, 81817600) 0 0 00000080:00200000:1.0:1789760796.646720:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.646737:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19974, 19974] 00020000:00200000:1.0:1789760796.646746:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81813504 chunk: [81813504, 81817600) 81817600, 4096 00020000:00200000:1.0:1789760796.646751:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.646769:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19974, 19974] 00000008:00010000:1.0:1789760796.646789:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.646798:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.646812:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81813504, 81817600) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.646827:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81813504=0x4e06000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.646892:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.646895:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81817600 00000020:00010000:1.0:1789760796.646904:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.646907:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.646911:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.646914:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.646918:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.646925:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.646933:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.646951:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760796.647097:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 81813504<>81817600 00000100:00100000:0.0:1789760796.647703:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503946880:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.663600:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987281 gen 2 00000008:00010000:0.0:1789760796.663654:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58a80 00000020:00200000:0.0:1789760796.663662:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.663673:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.663701:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58a80 00000100:00100000:0.0:1789760796.663736:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503946880:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.663877:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19974, 19974] 00000080:00200000:1.0:1789760796.663894:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.663900:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.663916:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81817600 i_blocks 159824 00000020:00200000:1.0:1789760796.663920:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.663930:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.663965:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81817600, count: 4096 00000080:00200000:1.0:1789760796.663973:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81817600, bytes: 4096 00000080:00200000:1.0:1789760796.663985:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.663995:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.663998:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81817600, 81821696) 0 0 00000080:00200000:1.0:1789760796.664002:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.664019:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19975, 19975] 00020000:00200000:1.0:1789760796.664028:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81817600 chunk: [81817600, 81821696) 81821696, 4096 00020000:00200000:1.0:1789760796.664065:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.664087:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19975, 19975] 00000008:00010000:1.0:1789760796.664106:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.664113:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.664125:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81817600, 81821696) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.664152:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81817600=0x4e07000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.664217:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.664220:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81821696 00000020:00010000:1.0:1789760796.664228:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.664231:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.664234:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.664237:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.664240:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.664250:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.664257:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.664342:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760796.664455:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 81817600<>81821696 00000100:00100000:0.0:1789760796.664841:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503947008:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.676151:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987282 gen 2 00000008:00010000:0.0:1789760796.676208:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58540 00000020:00200000:0.0:1789760796.676216:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.676229:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.676253:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58540 00000100:00100000:0.0:1789760796.676619:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503947008:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.676758:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19975, 19975] 00000080:00200000:1.0:1789760796.676776:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.676782:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.676798:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81821696 i_blocks 159832 00000020:00200000:1.0:1789760796.676802:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.676813:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.676847:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81821696, count: 4096 00000080:00200000:1.0:1789760796.676855:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81821696, bytes: 4096 00000080:00200000:1.0:1789760796.676866:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.676875:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.676877:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81821696, 81825792) 0 0 00000080:00200000:1.0:1789760796.676881:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.676898:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19976, 19976] 00020000:00200000:1.0:1789760796.676905:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81821696 chunk: [81821696, 81825792) 81825792, 4096 00020000:00200000:1.0:1789760796.676910:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.676928:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19976, 19976] 00000008:00010000:1.0:1789760796.676945:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.676952:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.676964:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81821696, 81825792) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.676977:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81821696=0x4e08000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.677146:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.677149:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81825792 00000020:00010000:1.0:1789760796.677159:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.677163:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.677167:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.677171:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.677174:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.677183:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.677189:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.677209:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58c00 00000008:00100000:1.0:1789760796.677486:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 81821696<>81825792 00000100:00100000:0.0:1789760796.677624:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503947136:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.688344:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987283 gen 2 00000008:00010000:0.0:1789760796.688394:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58c00 00000020:00200000:0.0:1789760796.688403:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.688426:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.688450:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58c00 00000100:00100000:0.0:1789760796.688480:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503947136:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.688856:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19976, 19976] 00000080:00200000:1.0:1789760796.688872:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.688878:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.688895:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81825792 i_blocks 159840 00000020:00200000:1.0:1789760796.688899:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.688910:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.688946:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81825792, count: 4096 00000080:00200000:1.0:1789760796.688953:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81825792, bytes: 4096 00000080:00200000:1.0:1789760796.688965:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.688973:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.688975:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81825792, 81829888) 0 0 00000080:00200000:1.0:1789760796.688980:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.688997:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19977, 19977] 00020000:00200000:1.0:1789760796.689006:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81825792 chunk: [81825792, 81829888) 81829888, 4096 00020000:00200000:1.0:1789760796.689010:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.689029:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19977, 19977] 00000008:00010000:1.0:1789760796.689082:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.689090:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.689103:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81825792, 81829888) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.689117:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81825792=0x4e09000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.689175:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.689178:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81829888 00000020:00010000:1.0:1789760796.689186:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.689190:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.689193:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.689197:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.689200:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.689208:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.689215:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.689232:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760796.689337:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 81825792<>81829888 00000100:00100000:0.0:1789760796.689418:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503947264:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.698918:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987284 gen 2 00000008:00010000:0.0:1789760796.698969:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d583c0 00000020:00200000:0.0:1789760796.698977:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.698989:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.699017:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d583c0 00000100:00100000:0.0:1789760796.699091:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503947264:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.699200:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19977, 19977] 00000080:00200000:2.0:1789760796.699217:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.699223:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.699238:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81829888 i_blocks 159848 00000020:00200000:2.0:1789760796.699242:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.699250:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.699296:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81829888, count: 4096 00000080:00200000:2.0:1789760796.699303:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81829888, bytes: 4096 00000080:00200000:2.0:1789760796.699314:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.699323:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.699325:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81829888, 81833984) 0 0 00000080:00200000:2.0:1789760796.699330:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.699346:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19978, 19978] 00020000:00200000:2.0:1789760796.699364:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81829888 chunk: [81829888, 81833984) 81833984, 4096 00020000:00200000:2.0:1789760796.699369:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.699390:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19978, 19978] 00000008:00010000:2.0:1789760796.699408:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.699414:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.699425:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81829888, 81833984) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.699438:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81829888=0x4e0a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.699498:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.699501:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81833984 00000020:00010000:2.0:1789760796.699510:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.699513:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.699516:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.699520:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.699522:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.699531:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.699538:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.699565:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000008:00100000:2.0:1789760796.699835:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27480 - object 0x280000401:2 offset 81829888<>81833984 00000100:00100000:2.0:1789760796.699882:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760796.699899:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503947392:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.710999:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987285 gen 2 00000008:00010000:3.0:1789760796.711098:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760796.711107:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.711118:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.711142:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760796.711171:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503947392:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.711266:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19978, 19978] 00000080:00200000:2.0:1789760796.711293:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.711299:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.711316:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81833984 i_blocks 159856 00000020:00200000:2.0:1789760796.711319:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.711329:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.711365:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81833984, count: 4096 00000080:00200000:2.0:1789760796.711373:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81833984, bytes: 4096 00000080:00200000:2.0:1789760796.711382:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.711391:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.711394:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81833984, 81838080) 0 0 00000080:00200000:2.0:1789760796.711398:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.711417:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19979, 19979] 00020000:00200000:2.0:1789760796.711426:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81833984 chunk: [81833984, 81838080) 81838080, 4096 00020000:00200000:2.0:1789760796.711431:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.711449:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19979, 19979] 00000008:00010000:2.0:1789760796.711466:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.711472:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.711482:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81833984, 81838080) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.711495:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81833984=0x4e0b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.711559:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.711563:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81838080 00000020:00010000:2.0:1789760796.711571:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.711575:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.711578:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.711582:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.711586:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.711905:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.711914:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.711933:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c540 00000008:00100000:2.0:1789760796.712067:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24e00 - object 0x280000401:2 offset 81833984<>81838080 00000100:00100000:3.0:1789760796.712134:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503947520:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.725487:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987286 gen 2 00000008:00010000:3.0:1789760796.725535:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760796.725543:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.725565:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.725773:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760796.725812:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503947520:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.727412:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19979, 19979] 00000080:00200000:2.0:1789760796.727436:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.727443:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.727462:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81838080 i_blocks 159864 00000020:00200000:2.0:1789760796.727467:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.727481:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.727515:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81838080, count: 4096 00000080:00200000:2.0:1789760796.727523:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81838080, bytes: 4096 00000080:00200000:2.0:1789760796.727538:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.727547:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.727549:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81838080, 81842176) 0 0 00000080:00200000:2.0:1789760796.727565:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.727582:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19980, 19980] 00020000:00200000:2.0:1789760796.730103:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81838080 chunk: [81838080, 81842176) 81842176, 4096 00020000:00200000:2.0:1789760796.730117:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.730153:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19980, 19980] 00000008:00010000:2.0:1789760796.730181:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.730187:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.730201:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81838080, 81842176) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.730217:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81838080=0x4e0c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.730815:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.730820:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81842176 00000020:00010000:2.0:1789760796.730830:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.730835:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.730838:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.730842:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.730846:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.730859:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.730867:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.730898:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c600 00000008:00100000:2.0:1789760796.731015:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 81838080<>81842176 00000100:00100000:3.0:1789760796.731418:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503947648:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.746243:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987287 gen 2 00000008:00010000:2.0:1789760796.746460:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:2.0:1789760796.746469:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.746480:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.746560:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:2.0:1789760796.746594:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503947648:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760796.748539:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19980, 19980] 00000080:00200000:0.0:1789760796.748826:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.748833:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.748851:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81842176 i_blocks 159872 00000020:00200000:0.0:1789760796.748856:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.748871:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.748959:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81842176, count: 4096 00000080:00200000:0.0:1789760796.748969:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81842176, bytes: 4096 00000080:00200000:0.0:1789760796.748983:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.748992:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760796.748994:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81842176, 81846272) 0 0 00000080:00200000:0.0:1789760796.748999:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.749023:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19981, 19981] 00020000:00200000:0.0:1789760796.749031:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81842176 chunk: [81842176, 81846272) 81846272, 4096 00020000:00200000:0.0:1789760796.749127:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.749150:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19981, 19981] 00000008:00010000:0.0:1789760796.749169:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.749176:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760796.749188:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81842176, 81846272) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.749204:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81842176=0x4e0d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.749352:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.749356:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81846272 00000020:00010000:0.0:1789760796.749366:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.749370:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.749374:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.749378:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.749381:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.749390:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.749397:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.749428:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760796.749539:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9bb80 - object 0x280000401:2 offset 81842176<>81846272 00000100:00100000:0.0:1789760796.749816:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760796.749844:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503947776:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.760947:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987288 gen 2 00000008:00010000:0.0:1789760796.760999:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:0.0:1789760796.761008:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.761020:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.761082:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:0.0:1789760796.761112:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503947776:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760796.761293:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19981, 19981] 00000080:00200000:1.0:1789760796.761310:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.761317:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.761334:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81846272 i_blocks 159880 00000020:00200000:1.0:1789760796.761338:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.761348:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.761383:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81846272, count: 4096 00000080:00200000:1.0:1789760796.761390:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81846272, bytes: 4096 00000080:00200000:1.0:1789760796.761401:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.761410:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760796.761412:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81846272, 81850368) 0 0 00000080:00200000:1.0:1789760796.761417:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.761432:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19982, 19982] 00020000:00200000:1.0:1789760796.761442:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81846272 chunk: [81846272, 81850368) 81850368, 4096 00020000:00200000:1.0:1789760796.761447:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.761465:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19982, 19982] 00000008:00010000:1.0:1789760796.761481:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.761488:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760796.761500:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81846272, 81850368) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.761514:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81846272=0x4e0e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.761580:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.761583:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81850368 00000020:00010000:1.0:1789760796.761601:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.761604:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.761608:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.761611:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.761615:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.761623:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.761630:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.761647:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760796.761755:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 81846272<>81850368 00000100:00100000:0.0:1789760796.761878:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503947904:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.773512:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987289 gen 2 00000008:00010000:0.0:1789760796.773575:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:0.0:1789760796.773585:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.773597:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.773627:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:0.0:1789760796.773661:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503947904:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.773768:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19982, 19982] 00000080:00200000:2.0:1789760796.773795:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.773801:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.773825:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81850368 i_blocks 159888 00000020:00200000:2.0:1789760796.773829:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.773839:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.773876:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81850368, count: 4096 00000080:00200000:2.0:1789760796.773882:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81850368, bytes: 4096 00000080:00200000:2.0:1789760796.773891:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.773900:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.773902:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81850368, 81854464) 0 0 00000080:00200000:2.0:1789760796.773906:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.773922:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19983, 19983] 00020000:00200000:2.0:1789760796.773931:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81850368 chunk: [81850368, 81854464) 81854464, 4096 00020000:00200000:2.0:1789760796.773935:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.773953:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19983, 19983] 00000008:00010000:2.0:1789760796.773973:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.773979:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.773990:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81850368, 81854464) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.774005:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81850368=0x4e0f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.774117:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.774120:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81854464 00000020:00010000:2.0:1789760796.774128:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.774131:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.774134:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.774137:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.774140:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.774148:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.774154:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.774170:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c600 00000008:00100000:2.0:1789760796.774264:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 81850368<>81854464 00000100:00100000:2.0:1789760796.774304:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760796.774320:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503948032:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.798395:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987290 gen 2 00000008:00010000:3.0:1789760796.798437:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760796.798445:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.798454:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.798512:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760796.798543:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503948032:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.798766:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19983, 19983] 00000080:00200000:2.0:1789760796.798781:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.798785:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.798800:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81854464 i_blocks 159896 00000020:00200000:2.0:1789760796.798803:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.798811:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.798847:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81854464, count: 4096 00000080:00200000:2.0:1789760796.798854:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81854464, bytes: 4096 00000080:00200000:2.0:1789760796.798864:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.798872:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.798874:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81854464, 81858560) 0 0 00000080:00200000:2.0:1789760796.798879:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.798896:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19984, 19984] 00020000:00200000:2.0:1789760796.798905:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81854464 chunk: [81854464, 81858560) 81858560, 4096 00020000:00200000:2.0:1789760796.798910:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.798928:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19984, 19984] 00000008:00010000:2.0:1789760796.798946:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.798953:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.798964:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81854464, 81858560) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.798979:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81854464=0x4e10000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.799111:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.799114:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81858560 00000020:00010000:2.0:1789760796.799122:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.799124:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.799127:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.799129:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.799131:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.799138:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.799143:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.799158:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cb40 00000008:00100000:2.0:1789760796.799252:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24700 - object 0x280000401:2 offset 81854464<>81858560 00000100:00100000:3.0:1789760796.799349:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503948160:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.813268:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987291 gen 2 00000008:00010000:3.0:1789760796.813327:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760796.813336:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.813347:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.813368:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760796.813399:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503948160:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.813797:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19984, 19984] 00000080:00200000:2.0:1789760796.813867:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.813874:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.813894:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81858560 i_blocks 159904 00000020:00200000:2.0:1789760796.813898:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.813907:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.813943:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81858560, count: 4096 00000080:00200000:2.0:1789760796.813950:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81858560, bytes: 4096 00000080:00200000:2.0:1789760796.813961:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.813970:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.813972:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81858560, 81862656) 0 0 00000080:00200000:2.0:1789760796.813976:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.813992:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19985, 19985] 00020000:00200000:2.0:1789760796.814001:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81858560 chunk: [81858560, 81862656) 81862656, 4096 00020000:00200000:2.0:1789760796.814007:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.814027:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19985, 19985] 00000008:00010000:2.0:1789760796.814083:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.814091:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.814104:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81858560, 81862656) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.814939:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81858560=0x4e11000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.815018:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.815021:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81862656 00000020:00010000:2.0:1789760796.815031:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.815078:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.815082:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.815086:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.815089:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.815100:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.815108:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.815584:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000008:00100000:2.0:1789760796.815728:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24380 - object 0x280000401:2 offset 81858560<>81862656 00000100:00100000:2.0:1789760796.815777:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760796.815806:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503948288:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.830271:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987292 gen 2 00000008:00010000:3.0:1789760796.830326:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760796.830335:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.830347:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.830436:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760796.830470:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503948288:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760796.830666:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19985, 19985] 00000080:00200000:0.0:1789760796.830683:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.830690:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.830705:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81862656 i_blocks 159912 00000020:00200000:0.0:1789760796.830709:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.830721:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.830758:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81862656, count: 4096 00000080:00200000:0.0:1789760796.830764:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81862656, bytes: 4096 00000080:00200000:0.0:1789760796.830777:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.830787:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760796.830789:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81862656, 81866752) 0 0 00000080:00200000:0.0:1789760796.830794:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.830815:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19986, 19986] 00020000:00200000:0.0:1789760796.830825:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81862656 chunk: [81862656, 81866752) 81866752, 4096 00020000:00200000:0.0:1789760796.830830:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.830850:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19986, 19986] 00000008:00010000:0.0:1789760796.830869:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.830876:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760796.830887:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81862656, 81866752) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.830902:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81862656=0x4e12000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.830969:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.830973:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81866752 00000020:00010000:0.0:1789760796.830981:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.830984:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.830987:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.830990:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.830993:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.831002:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.831009:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.831025:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760796.831155:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9bb80 - object 0x280000401:2 offset 81862656<>81866752 00000100:00100000:0.0:1789760796.831316:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760796.831335:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503948416:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760796.843394:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987293 gen 2 00000008:00010000:0.0:1789760796.843447:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:0.0:1789760796.843457:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.843470:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.843491:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:0.0:1789760796.843525:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503948416:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760796.844099:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19986, 19986] 00000080:00200000:3.0:1789760796.844116:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.844121:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.844138:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81866752 i_blocks 159920 00000020:00200000:3.0:1789760796.844142:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.844157:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.844193:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81866752, count: 4096 00000080:00200000:3.0:1789760796.844200:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81866752, bytes: 4096 00000080:00200000:3.0:1789760796.844211:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.844220:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760796.844222:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81866752, 81870848) 0 0 00000080:00200000:3.0:1789760796.844226:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.844243:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19987, 19987] 00020000:00200000:3.0:1789760796.844252:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81866752 chunk: [81866752, 81870848) 81870848, 4096 00020000:00200000:3.0:1789760796.844257:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.845124:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19987, 19987] 00000008:00010000:3.0:1789760796.845155:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.845161:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:3.0:1789760796.845174:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81866752, 81870848) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.845190:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81866752=0x4e13000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.845259:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.845262:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81870848 00000020:00010000:3.0:1789760796.845272:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.845275:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.845278:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.845282:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.845285:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.845294:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.845302:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.845331:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760796.845444:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070fb80 - object 0x280000401:2 offset 81866752<>81870848 00000100:00100000:2.0:1789760796.845530:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760796.845568:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503948544:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.864955:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987294 gen 2 00000008:00010000:3.0:1789760796.865006:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760796.865016:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.865027:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.865394:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000080:00200000:2.0:1789760796.865398:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19987, 19987] 00000080:00200000:2.0:1789760796.865415:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.865420:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760796.865428:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503948544:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.865435:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81870848 i_blocks 159928 00000020:00200000:2.0:1789760796.865439:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.865449:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.865480:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81870848, count: 4096 00000080:00200000:2.0:1789760796.865486:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81870848, bytes: 4096 00000080:00200000:2.0:1789760796.865495:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.865631:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.865635:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81870848, 81874944) 0 0 00000080:00200000:2.0:1789760796.865639:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.865653:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19988, 19988] 00020000:00200000:2.0:1789760796.865661:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81870848 chunk: [81870848, 81874944) 81874944, 4096 00020000:00200000:2.0:1789760796.865665:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.865681:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19988, 19988] 00000008:00010000:2.0:1789760796.865696:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.865701:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.865710:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81870848, 81874944) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.865721:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81870848=0x4e14000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.865771:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.865774:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81874944 00000020:00010000:2.0:1789760796.865780:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.865783:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.865785:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.865787:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.865789:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.865795:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.865800:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.865814:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c900 00000008:00100000:2.0:1789760796.865918:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25c00 - object 0x280000401:2 offset 81870848<>81874944 00000100:00100000:3.0:1789760796.866446:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760796.866469:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503948672:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.889601:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987295 gen 2 00000008:00010000:3.0:1789760796.889652:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760796.889661:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.889671:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.889695:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760796.889727:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503948672:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.889830:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19988, 19988] 00000080:00200000:2.0:1789760796.889848:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.889854:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.889869:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81874944 i_blocks 159936 00000020:00200000:2.0:1789760796.889873:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.889882:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.889918:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81874944, count: 4096 00000080:00200000:2.0:1789760796.889925:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81874944, bytes: 4096 00000080:00200000:2.0:1789760796.889934:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.889943:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.889945:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81874944, 81879040) 0 0 00000080:00200000:2.0:1789760796.889950:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.889966:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19989, 19989] 00020000:00200000:2.0:1789760796.889977:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81874944 chunk: [81874944, 81879040) 81879040, 4096 00020000:00200000:2.0:1789760796.889982:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.890002:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19989, 19989] 00000008:00010000:2.0:1789760796.890020:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.890026:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.890074:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81874944, 81879040) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.890089:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81874944=0x4e15000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.890149:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.890152:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81879040 00000020:00010000:2.0:1789760796.890161:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.890165:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.890167:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.890171:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.890173:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.890181:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.890189:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.890204:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c240 00000008:00100000:2.0:1789760796.890310:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25180 - object 0x280000401:2 offset 81874944<>81879040 00000100:00100000:3.0:1789760796.890607:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503948800:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.906173:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987296 gen 2 00000008:00010000:3.0:1789760796.906218:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760796.906226:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.906235:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.906287:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760796.906313:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503948800:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.907305:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19989, 19989] 00000080:00200000:2.0:1789760796.907327:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.907332:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.907350:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81879040 i_blocks 159944 00000020:00200000:2.0:1789760796.907355:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.907367:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.907401:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81879040, count: 4096 00000080:00200000:2.0:1789760796.907409:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81879040, bytes: 4096 00000080:00200000:2.0:1789760796.907421:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.907430:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.907432:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81879040, 81883136) 0 0 00000080:00200000:2.0:1789760796.907437:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.907452:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19990, 19990] 00020000:00200000:2.0:1789760796.907462:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81879040 chunk: [81879040, 81883136) 81883136, 4096 00020000:00200000:2.0:1789760796.907466:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.907485:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19990, 19990] 00000008:00010000:2.0:1789760796.907503:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.907510:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.907522:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81879040, 81883136) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.907648:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81879040=0x4e16000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.907719:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.907722:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81883136 00000020:00010000:2.0:1789760796.907731:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.907734:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.907737:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.907741:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.907743:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.907752:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.907759:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.907785:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760796.907974:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27800 - object 0x280000401:2 offset 81879040<>81883136 00000100:00100000:3.0:1789760796.908077:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503948928:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.924996:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987297 gen 2 00000008:00010000:3.0:1789760796.925066:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760796.925075:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.925091:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.925120:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760796.925157:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503948928:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.925219:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19990, 19990] 00000080:00200000:2.0:1789760796.925237:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.925244:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.925260:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81883136 i_blocks 159952 00000020:00200000:2.0:1789760796.925264:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.925276:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.925314:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81883136, count: 4096 00000080:00200000:2.0:1789760796.925321:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81883136, bytes: 4096 00000080:00200000:2.0:1789760796.925337:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.925346:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.925348:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81883136, 81887232) 0 0 00000080:00200000:2.0:1789760796.925352:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.925369:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19991, 19991] 00020000:00200000:2.0:1789760796.925378:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81883136 chunk: [81883136, 81887232) 81887232, 4096 00020000:00200000:2.0:1789760796.925384:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.925406:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19991, 19991] 00000008:00010000:2.0:1789760796.925434:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.925441:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.925453:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81883136, 81887232) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.925469:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81883136=0x4e17000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.925528:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.925531:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81887232 00000020:00010000:2.0:1789760796.925540:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.925543:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.925547:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.925560:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.925564:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.925574:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.925582:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.925599:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ce40 00000008:00100000:2.0:1789760796.925706:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26a00 - object 0x280000401:2 offset 81883136<>81887232 00000100:00100000:3.0:1789760796.925814:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949056:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.936846:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987298 gen 2 00000008:00010000:3.0:1789760796.936897:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760796.936905:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.936917:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.936942:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760796.936977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949056:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.938174:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19991, 19991] 00000080:00200000:2.0:1789760796.938200:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.938213:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.938232:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81887232 i_blocks 159960 00000020:00200000:2.0:1789760796.938238:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.938253:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.938288:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81887232, count: 4096 00000080:00200000:2.0:1789760796.938296:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81887232, bytes: 4096 00000080:00200000:2.0:1789760796.938308:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.938317:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.938319:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81887232, 81891328) 0 0 00000080:00200000:2.0:1789760796.938324:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.938341:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19992, 19992] 00020000:00200000:2.0:1789760796.938351:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81887232 chunk: [81887232, 81891328) 81891328, 4096 00020000:00200000:2.0:1789760796.938356:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.938376:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19992, 19992] 00000008:00010000:2.0:1789760796.938396:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.938402:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.938415:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81887232, 81891328) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.938431:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81887232=0x4e18000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.938502:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.938505:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81891328 00000020:00010000:2.0:1789760796.938513:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.938517:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.938520:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.938524:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.938527:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.938536:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.938543:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.938575:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c000 00000008:00100000:2.0:1789760796.938690:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25f80 - object 0x280000401:2 offset 81887232<>81891328 00000100:00100000:3.0:1789760796.938986:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760796.939015:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949184:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.948923:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987299 gen 2 00000008:00010000:3.0:1789760796.948975:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760796.948983:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.948993:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.949109:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760796.949138:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949184:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.949642:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19992, 19992] 00000080:00200000:2.0:1789760796.949660:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.949666:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.949683:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81891328 i_blocks 159968 00000020:00200000:2.0:1789760796.949686:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.949697:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.949734:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81891328, count: 4096 00000080:00200000:2.0:1789760796.949742:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81891328, bytes: 4096 00000080:00200000:2.0:1789760796.949752:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.949761:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.949763:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81891328, 81895424) 0 0 00000080:00200000:2.0:1789760796.949767:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.949783:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19993, 19993] 00020000:00200000:2.0:1789760796.949792:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81891328 chunk: [81891328, 81895424) 81895424, 4096 00020000:00200000:2.0:1789760796.949797:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.949816:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19993, 19993] 00000008:00010000:2.0:1789760796.949835:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.949842:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.949854:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81891328, 81895424) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.949894:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81891328=0x4e19000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.949960:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.949963:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81895424 00000020:00010000:2.0:1789760796.949972:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.949975:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.949977:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.949980:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.949982:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.949989:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.949996:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.950014:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cf00 00000008:00100000:2.0:1789760796.950128:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26300 - object 0x280000401:2 offset 81891328<>81895424 00000100:00100000:3.0:1789760796.950183:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949312:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.959805:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987300 gen 2 00000008:00010000:3.0:1789760796.959914:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760796.959924:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.959936:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.960018:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760796.960097:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949312:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.960195:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19993, 19993] 00000080:00200000:2.0:1789760796.960212:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.960219:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.960235:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81895424 i_blocks 159976 00000020:00200000:2.0:1789760796.960239:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.960248:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.960358:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81895424, count: 4096 00000080:00200000:2.0:1789760796.960366:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81895424, bytes: 4096 00000080:00200000:2.0:1789760796.960377:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.960385:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.960387:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81895424, 81899520) 0 0 00000080:00200000:2.0:1789760796.960392:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.960408:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19994, 19994] 00020000:00200000:2.0:1789760796.960417:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81895424 chunk: [81895424, 81899520) 81899520, 4096 00020000:00200000:2.0:1789760796.960423:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.960443:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19994, 19994] 00000008:00010000:2.0:1789760796.960463:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.960469:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.960481:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81895424, 81899520) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.960496:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81895424=0x4e1a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.960566:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.960569:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81899520 00000020:00010000:2.0:1789760796.960578:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.960581:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.960584:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.960588:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.960591:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.960600:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.960606:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.960623:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cc00 00000008:00100000:2.0:1789760796.960736:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24a80 - object 0x280000401:2 offset 81895424<>81899520 00000100:00100000:3.0:1789760796.960811:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503949440:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760796.976623:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987301 gen 2 00000008:00010000:3.0:1789760796.976687:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760796.976696:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.976707:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.976731:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760796.976772:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503949440:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.976955:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19994, 19994] 00000080:00200000:2.0:1789760796.976972:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.976978:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.976994:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81899520 i_blocks 159984 00000020:00200000:2.0:1789760796.976998:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.977008:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.977086:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81899520, count: 4096 00000080:00200000:2.0:1789760796.977093:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81899520, bytes: 4096 00000080:00200000:2.0:1789760796.977103:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.977112:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.977115:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81899520, 81903616) 0 0 00000080:00200000:2.0:1789760796.977120:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.977136:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19995, 19995] 00020000:00200000:2.0:1789760796.977145:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81899520 chunk: [81899520, 81903616) 81903616, 4096 00020000:00200000:2.0:1789760796.977150:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.977169:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19995, 19995] 00000008:00010000:2.0:1789760796.977186:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.977192:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.977204:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81899520, 81903616) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.977219:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81899520=0x4e1b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.977279:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.977282:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81903616 00000020:00010000:2.0:1789760796.977291:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.977294:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.977297:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.977300:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.977303:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.977311:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.977318:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.977332:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c300 00000008:00100000:2.0:1789760796.977436:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27100 - object 0x280000401:2 offset 81899520<>81903616 00000100:00100000:3.0:1789760796.977983:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949568:192.168.203.158@tcp:4:dd.0 00010000:00010000:3.0:1789760796.990509:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760796.995396:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987302 gen 2 00000008:00010000:3.0:1789760796.995450:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760796.995459:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.995471:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.997537:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760796.997609:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949568:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760796.999031:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19995, 19995] 00000080:00200000:2.0:1789760796.999111:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.999119:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.999138:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81903616 i_blocks 159992 00000020:00200000:2.0:1789760796.999143:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.999157:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.999194:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81903616, count: 4096 00000080:00200000:2.0:1789760796.999203:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81903616, bytes: 4096 00000080:00200000:2.0:1789760796.999219:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.999228:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760796.999230:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81903616, 81907712) 0 0 00000080:00200000:2.0:1789760796.999235:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.999252:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19996, 19996] 00020000:00200000:2.0:1789760796.999264:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81903616 chunk: [81903616, 81907712) 81907712, 4096 00020000:00200000:2.0:1789760796.999268:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.999291:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19996, 19996] 00000008:00010000:2.0:1789760796.999310:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.999316:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760796.999328:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81903616, 81907712) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.999343:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81903616=0x4e1c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.999407:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.999411:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81907712 00000020:00010000:2.0:1789760796.999419:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.999423:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.999426:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.999429:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.999432:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.999442:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.999448:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.999469:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000008:00100000:2.0:1789760796.999596:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27100 - object 0x280000401:2 offset 81903616<>81907712 00000100:00100000:3.0:1789760797.000454:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760797.000486:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949696:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.021845:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987303 gen 2 00000008:00010000:3.0:1789760797.021895:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760797.021904:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.021916:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.021978:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760797.022010:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949696:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.022594:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19996, 19996] 00000080:00200000:2.0:1789760797.022611:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.022617:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.022824:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81907712 i_blocks 160000 00000020:00200000:2.0:1789760797.022830:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.022841:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.022877:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81907712, count: 4096 00000080:00200000:2.0:1789760797.022884:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81907712, bytes: 4096 00000080:00200000:2.0:1789760797.022897:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.022905:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.022908:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81907712, 81911808) 0 0 00000080:00200000:2.0:1789760797.022912:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.022928:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19997, 19997] 00020000:00200000:2.0:1789760797.022937:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81907712 chunk: [81907712, 81911808) 81911808, 4096 00020000:00200000:2.0:1789760797.022955:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.022976:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19997, 19997] 00000008:00010000:2.0:1789760797.022994:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.023000:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.023013:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81907712, 81911808) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.023029:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81907712=0x4e1d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.023134:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.023137:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81911808 00000020:00010000:2.0:1789760797.023146:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.023150:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.023153:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.023156:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.023159:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.023167:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.023174:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.023186:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000008:00100000:2.0:1789760797.023909:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24a80 - object 0x280000401:2 offset 81907712<>81911808 00000100:00100000:3.0:1789760797.024583:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949824:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.038218:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987304 gen 2 00000008:00010000:3.0:1789760797.038263:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760797.038270:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.038282:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.038303:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760797.038334:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949824:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.038560:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19997, 19997] 00000080:00200000:2.0:1789760797.038576:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.038583:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.038613:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81911808 i_blocks 160008 00000020:00200000:2.0:1789760797.038618:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.038627:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.038661:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81911808, count: 4096 00000080:00200000:2.0:1789760797.038669:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81911808, bytes: 4096 00000080:00200000:2.0:1789760797.038679:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.038688:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.038690:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81911808, 81915904) 0 0 00000080:00200000:2.0:1789760797.038695:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.038711:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19998, 19998] 00020000:00200000:2.0:1789760797.038720:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81911808 chunk: [81911808, 81915904) 81915904, 4096 00020000:00200000:2.0:1789760797.038725:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.038744:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19998, 19998] 00000008:00010000:2.0:1789760797.038762:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.038768:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.038780:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81911808, 81915904) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.038795:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81911808=0x4e1e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.038856:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.038859:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81915904 00000020:00010000:2.0:1789760797.038869:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.038873:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.038876:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.038880:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.038883:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.038892:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.038899:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.038916:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c480 00000008:00100000:2.0:1789760797.039024:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26300 - object 0x280000401:2 offset 81911808<>81915904 00000100:00100000:3.0:1789760797.039352:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760797.039384:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949952:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.052797:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987305 gen 2 00000008:00010000:3.0:1789760797.052851:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760797.052984:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.052995:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.053145:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760797.053176:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503949952:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760797.053366:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19998, 19998] 00000080:00200000:0.0:1789760797.055770:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.055784:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.056267:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81915904 i_blocks 160016 00000020:00200000:0.0:1789760797.056485:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.056503:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.056537:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81915904, count: 4096 00000080:00200000:0.0:1789760797.056546:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81915904, bytes: 4096 00000080:00200000:0.0:1789760797.056572:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.056582:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760797.056584:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81915904, 81920000) 0 0 00000080:00200000:0.0:1789760797.056991:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.057015:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [19999, 19999] 00020000:00200000:0.0:1789760797.057026:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81915904 chunk: [81915904, 81920000) 81920000, 4096 00020000:00200000:0.0:1789760797.057031:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.057089:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [19999, 19999] 00000008:00010000:0.0:1789760797.057110:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.057116:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760797.057129:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81915904, 81920000) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.057146:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81915904=0x4e1f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.057222:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.057225:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81920000 00000020:00010000:0.0:1789760797.057234:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.057238:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.057242:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.057245:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.057249:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.057259:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.057266:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.057508:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760797.057707:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9bb80 - object 0x280000401:2 offset 81915904<>81920000 00000100:00100000:1.0:1789760797.057822:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503950080:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760797.092088:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987306 gen 2 00000008:00010000:1.0:1789760797.092149:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760797.092158:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.092170:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.092193:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760797.092226:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503950080:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.092337:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [19999, 19999] 00000080:00200000:2.0:1789760797.092354:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.092361:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.092377:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81920000 i_blocks 160024 00000020:00200000:2.0:1789760797.092381:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.092391:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.092426:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81920000, count: 4096 00000080:00200000:2.0:1789760797.092434:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81920000, bytes: 4096 00000080:00200000:2.0:1789760797.092444:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.092453:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.092455:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81920000, 81924096) 0 0 00000080:00200000:2.0:1789760797.092460:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.092475:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20000, 20000] 00020000:00200000:2.0:1789760797.092485:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81920000 chunk: [81920000, 81924096) 81924096, 4096 00020000:00200000:2.0:1789760797.092489:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.092508:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20000, 20000] 00000008:00010000:2.0:1789760797.092525:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.092532:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.092543:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81920000, 81924096) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.092571:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81920000=0x4e20000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.092629:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.092632:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81924096 00000020:00010000:2.0:1789760797.092641:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.092645:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.092648:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.092651:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.092654:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.092662:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.092670:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.092687:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c480 00000008:00100000:2.0:1789760797.092788:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25f80 - object 0x280000401:2 offset 81920000<>81924096 00000100:00100000:2.0:1789760797.092838:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760797.092932:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503950208:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.123874:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987307 gen 2 00000008:00010000:3.0:1789760797.123930:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c480 00000020:00200000:3.0:1789760797.123940:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.123952:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.123976:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c480 00000100:00100000:3.0:1789760797.124008:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503950208:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.124265:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20000, 20000] 00000080:00200000:2.0:1789760797.124282:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.124288:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.124305:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81924096 i_blocks 160032 00000020:00200000:2.0:1789760797.124309:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.124319:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.124356:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81924096, count: 4096 00000080:00200000:2.0:1789760797.124364:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81924096, bytes: 4096 00000080:00200000:2.0:1789760797.124373:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.124382:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.124386:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81924096, 81928192) 0 0 00000080:00200000:2.0:1789760797.124390:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.124409:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20001, 20001] 00020000:00200000:2.0:1789760797.124418:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81924096 chunk: [81924096, 81928192) 81928192, 4096 00020000:00200000:2.0:1789760797.124423:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.124442:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20001, 20001] 00000008:00010000:2.0:1789760797.124799:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.124806:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.124819:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81924096, 81928192) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.124835:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81924096=0x4e21000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.124899:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.124902:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81928192 00000020:00010000:2.0:1789760797.124909:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.124913:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.124917:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.124921:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.124924:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.124932:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.124939:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.124960:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000008:00100000:2.0:1789760797.125094:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26a00 - object 0x280000401:2 offset 81924096<>81928192 00000100:00100000:3.0:1789760797.125165:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503950336:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.148959:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987308 gen 2 00000008:00010000:3.0:1789760797.149031:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000020:00200000:3.0:1789760797.149056:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.149067:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.149148:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c0c0 00000100:00100000:3.0:1789760797.149174:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503950336:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.153181:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20001, 20001] 00000080:00200000:2.0:1789760797.153210:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.153217:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.153237:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81928192 i_blocks 160040 00000020:00200000:2.0:1789760797.153243:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.153260:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.153297:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81928192, count: 4096 00000080:00200000:2.0:1789760797.153305:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81928192, bytes: 4096 00000080:00200000:2.0:1789760797.153320:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.153330:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.153333:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81928192, 81932288) 0 0 00000080:00200000:2.0:1789760797.153338:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.153355:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20002, 20002] 00020000:00200000:2.0:1789760797.153365:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81928192 chunk: [81928192, 81932288) 81932288, 4096 00020000:00200000:2.0:1789760797.153369:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.153389:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20002, 20002] 00000008:00010000:2.0:1789760797.153408:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.153414:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.153427:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81928192, 81932288) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.153443:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81928192=0x4e22000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.154100:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.154103:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81932288 00000020:00010000:2.0:1789760797.154114:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.154118:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.154121:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.154125:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.154128:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.154139:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.154146:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.154176:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000008:00100000:2.0:1789760797.154290:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27800 - object 0x280000401:2 offset 81928192<>81932288 00000100:00100000:3.0:1789760797.154682:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503950464:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.167798:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987309 gen 2 00000008:00010000:3.0:1789760797.167851:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000020:00200000:3.0:1789760797.167859:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.167869:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.167890:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ccc0 00000100:00100000:3.0:1789760797.167920:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503950464:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.168299:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20002, 20002] 00000080:00200000:2.0:1789760797.168314:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.168321:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.168336:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81932288 i_blocks 160048 00000020:00200000:2.0:1789760797.168339:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.168348:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.168383:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81932288, count: 4096 00000080:00200000:2.0:1789760797.168389:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81932288, bytes: 4096 00000080:00200000:2.0:1789760797.168400:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.168408:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.168411:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81932288, 81936384) 0 0 00000080:00200000:2.0:1789760797.168415:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.168431:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20003, 20003] 00020000:00200000:2.0:1789760797.168440:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81932288 chunk: [81932288, 81936384) 81936384, 4096 00020000:00200000:2.0:1789760797.168503:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.168525:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20003, 20003] 00000008:00010000:2.0:1789760797.168545:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.168560:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.168572:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81932288, 81936384) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.168963:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81932288=0x4e23000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.169521:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.169526:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81936384 00000020:00010000:2.0:1789760797.169538:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.169542:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.169546:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.169559:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.169562:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.169574:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.169583:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.169675:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c300 00000008:00100000:2.0:1789760797.169786:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25180 - object 0x280000401:2 offset 81932288<>81936384 00000100:00100000:3.0:1789760797.170051:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760797.170075:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503950592:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.184397:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987310 gen 2 00000008:00010000:3.0:1789760797.184449:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c300 00000020:00200000:3.0:1789760797.184459:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.184470:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.184628:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c300 00000100:00100000:3.0:1789760797.184660:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503950592:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760797.184891:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20003, 20003] 00000080:00200000:0.0:1789760797.184908:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.184914:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.184931:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81936384 i_blocks 160056 00000020:00200000:0.0:1789760797.184935:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.184945:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.184983:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81936384, count: 4096 00000080:00200000:0.0:1789760797.184990:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81936384, bytes: 4096 00000080:00200000:0.0:1789760797.185002:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.185010:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760797.185013:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81936384, 81940480) 0 0 00000080:00200000:0.0:1789760797.185018:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.186869:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20004, 20004] 00020000:00200000:0.0:1789760797.186890:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81936384 chunk: [81936384, 81940480) 81940480, 4096 00020000:00200000:0.0:1789760797.186896:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.186926:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20004, 20004] 00000008:00010000:0.0:1789760797.186950:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.186957:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760797.186970:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81936384, 81940480) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.186986:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81936384=0x4e24000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.187075:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.187079:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81940480 00000020:00010000:0.0:1789760797.187088:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.187092:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.187095:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.187099:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.187665:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.187683:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.187692:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.187724:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760797.187837:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99f80 - object 0x280000401:2 offset 81936384<>81940480 00000100:00100000:1.0:1789760797.187940:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503950720:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760797.206990:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987311 gen 2 00000008:00010000:1.0:1789760797.207073:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760797.207083:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.207095:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.207181:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760797.207212:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503950720:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.207531:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20004, 20004] 00000080:00200000:2.0:1789760797.207558:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.207565:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.207580:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81940480 i_blocks 160064 00000020:00200000:2.0:1789760797.207592:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.207604:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.207822:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81940480, count: 4096 00000080:00200000:2.0:1789760797.207831:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81940480, bytes: 4096 00000080:00200000:2.0:1789760797.207843:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.207851:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.207854:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81940480, 81944576) 0 0 00000080:00200000:2.0:1789760797.207858:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.207874:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20005, 20005] 00020000:00200000:2.0:1789760797.207883:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81940480 chunk: [81940480, 81944576) 81944576, 4096 00020000:00200000:2.0:1789760797.207889:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.207910:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20005, 20005] 00000008:00010000:2.0:1789760797.207929:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.207935:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.207946:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81940480, 81944576) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.207961:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81940480=0x4e25000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.208019:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.208022:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81944576 00000020:00010000:2.0:1789760797.208030:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.209256:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.209263:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.209689:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.209695:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.209709:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.209720:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.209743:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cc00 00000008:00100000:2.0:1789760797.209857:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25c00 - object 0x280000401:2 offset 81940480<>81944576 00000100:00100000:3.0:1789760797.210021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503950848:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.224025:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987312 gen 2 00000008:00010000:3.0:1789760797.224102:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cc00 00000020:00200000:3.0:1789760797.224184:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.224197:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.224220:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cc00 00000100:00100000:3.0:1789760797.224252:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503950848:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.224422:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20005, 20005] 00000080:00200000:2.0:1789760797.224444:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.224450:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.224468:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81944576 i_blocks 160072 00000020:00200000:2.0:1789760797.224472:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.224481:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.224520:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81944576, count: 4096 00000080:00200000:2.0:1789760797.224528:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81944576, bytes: 4096 00000080:00200000:2.0:1789760797.224541:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.224559:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.224562:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81944576, 81948672) 0 0 00000080:00200000:2.0:1789760797.224567:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.224584:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20006, 20006] 00020000:00200000:2.0:1789760797.224601:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81944576 chunk: [81944576, 81948672) 81948672, 4096 00020000:00200000:2.0:1789760797.224606:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.225405:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20006, 20006] 00000008:00010000:2.0:1789760797.225434:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.225442:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.225457:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81944576, 81948672) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.225472:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81944576=0x4e26000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.225563:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.225566:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81948672 00000020:00010000:2.0:1789760797.225574:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.225578:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.225581:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.225592:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.225596:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.225607:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.226389:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.226424:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cf00 00000008:00100000:2.0:1789760797.226556:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24380 - object 0x280000401:2 offset 81944576<>81948672 00000100:00100000:3.0:1789760797.227095:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503950976:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.244001:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987313 gen 2 00000008:00010000:3.0:1789760797.244084:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cf00 00000020:00200000:3.0:1789760797.244092:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.244103:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.244132:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cf00 00000100:00100000:3.0:1789760797.244161:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503950976:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.244465:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20006, 20006] 00000080:00200000:2.0:1789760797.244482:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.244488:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.244504:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81948672 i_blocks 160080 00000020:00200000:2.0:1789760797.244508:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.244518:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.244564:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81948672, count: 4096 00000080:00200000:2.0:1789760797.244572:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81948672, bytes: 4096 00000080:00200000:2.0:1789760797.244582:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.244590:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.244593:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81948672, 81952768) 0 0 00000080:00200000:2.0:1789760797.244597:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.245386:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20007, 20007] 00020000:00200000:2.0:1789760797.245400:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81948672 chunk: [81948672, 81952768) 81952768, 4096 00020000:00200000:2.0:1789760797.245405:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.245428:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20007, 20007] 00000008:00010000:2.0:1789760797.245450:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.245456:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.245468:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81948672, 81952768) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.245484:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81948672=0x4e27000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.245721:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.245725:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81952768 00000020:00010000:2.0:1789760797.245734:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.245738:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.245741:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.245745:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.245749:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.245758:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.245765:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.245791:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c000 00000008:00100000:2.0:1789760797.245900:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24700 - object 0x280000401:2 offset 81948672<>81952768 00000100:00100000:3.0:1789760797.247112:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503951104:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.259342:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987314 gen 2 00000008:00010000:3.0:1789760797.259378:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c000 00000020:00200000:3.0:1789760797.259384:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.259392:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.259416:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c000 00000100:00100000:3.0:1789760797.259440:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503951104:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.259559:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20007, 20007] 00000080:00200000:2.0:1789760797.259578:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.259594:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.259607:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81952768 i_blocks 160088 00000020:00200000:2.0:1789760797.259611:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.259619:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.259645:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81952768, count: 4096 00000080:00200000:2.0:1789760797.259650:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81952768, bytes: 4096 00000080:00200000:2.0:1789760797.259659:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.259666:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.259668:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81952768, 81956864) 0 0 00000080:00200000:2.0:1789760797.259672:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.259684:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20008, 20008] 00020000:00200000:2.0:1789760797.259690:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81952768 chunk: [81952768, 81956864) 81956864, 4096 00020000:00200000:2.0:1789760797.259694:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.259710:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20008, 20008] 00000008:00010000:2.0:1789760797.259725:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.259730:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.259739:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81952768, 81956864) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.259749:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81952768=0x4e28000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.259795:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.259798:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81956864 00000020:00010000:2.0:1789760797.259805:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.259808:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.259811:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.259814:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.259817:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.259824:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.259830:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.259844:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0ce40 00000008:00100000:2.0:1789760797.260077:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 81952768<>81956864 00000100:00100000:3.0:1789760797.260160:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760797.260174:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503951232:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.277461:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987315 gen 2 00000008:00010000:3.0:1789760797.277496:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0ce40 00000020:00200000:3.0:1789760797.277502:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.277511:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.277533:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0ce40 00000100:00100000:3.0:1789760797.277567:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503951232:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.278755:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20008, 20008] 00000080:00200000:2.0:1789760797.278771:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.278777:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.278792:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81956864 i_blocks 160096 00000020:00200000:2.0:1789760797.278796:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.278806:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.278835:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81956864, count: 4096 00000080:00200000:2.0:1789760797.278842:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81956864, bytes: 4096 00000080:00200000:2.0:1789760797.278853:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.278861:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.278864:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81956864, 81960960) 0 0 00000080:00200000:2.0:1789760797.278867:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.278881:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20009, 20009] 00020000:00200000:2.0:1789760797.278889:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81956864 chunk: [81956864, 81960960) 81960960, 4096 00020000:00200000:2.0:1789760797.278894:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.278910:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20009, 20009] 00000008:00010000:2.0:1789760797.278924:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.278930:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.278939:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81956864, 81960960) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.278950:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81956864=0x4e29000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.279009:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.279011:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81960960 00000020:00010000:2.0:1789760797.279018:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.279021:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.279024:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.279027:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.279029:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.279065:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.279072:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.279091:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000008:00100000:2.0:1789760797.279183:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24e00 - object 0x280000401:2 offset 81956864<>81960960 00000100:00100000:3.0:1789760797.279252:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503951360:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.296253:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987316 gen 2 00000008:00010000:3.0:1789760797.296292:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000020:00200000:3.0:1789760797.296299:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.296308:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.296326:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c6c0 00000100:00100000:3.0:1789760797.296348:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503951360:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.296441:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20009, 20009] 00000080:00200000:2.0:1789760797.296455:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.296460:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.296472:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81960960 i_blocks 160104 00000020:00200000:2.0:1789760797.296475:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.296483:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.296508:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81960960, count: 4096 00000080:00200000:2.0:1789760797.296514:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81960960, bytes: 4096 00000080:00200000:2.0:1789760797.296522:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.296529:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.296532:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81960960, 81965056) 0 0 00000080:00200000:2.0:1789760797.296535:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.296561:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20010, 20010] 00020000:00200000:2.0:1789760797.296568:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81960960 chunk: [81960960, 81965056) 81965056, 4096 00020000:00200000:2.0:1789760797.296572:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.296596:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20010, 20010] 00000008:00010000:2.0:1789760797.296611:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.296616:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.296625:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81960960, 81965056) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.296636:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81960960=0x4e2a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.296679:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.296682:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81965056 00000020:00010000:2.0:1789760797.296690:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.296693:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.296696:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.296699:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.296701:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.296708:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.296713:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.296727:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c240 00000008:00100000:2.0:1789760797.296807:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27480 - object 0x280000401:2 offset 81960960<>81965056 00000100:00100000:3.0:1789760797.297014:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503951488:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.317672:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987317 gen 2 00000008:00010000:3.0:1789760797.317707:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c240 00000020:00200000:3.0:1789760797.317714:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.317723:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.317747:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c240 00000100:00100000:3.0:1789760797.317771:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503951488:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.318371:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20010, 20010] 00000080:00200000:2.0:1789760797.318386:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.318392:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.318404:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81965056 i_blocks 160112 00000020:00200000:2.0:1789760797.318408:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.318417:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.318443:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81965056, count: 4096 00000080:00200000:2.0:1789760797.318450:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81965056, bytes: 4096 00000080:00200000:2.0:1789760797.318458:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.318466:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.318468:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81965056, 81969152) 0 0 00000080:00200000:2.0:1789760797.318472:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.318484:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20011, 20011] 00020000:00200000:2.0:1789760797.318491:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81965056 chunk: [81965056, 81969152) 81969152, 4096 00020000:00200000:2.0:1789760797.318495:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.318509:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20011, 20011] 00000008:00010000:2.0:1789760797.318523:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.318528:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.318538:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81965056, 81969152) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.318562:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81965056=0x4e2b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.318612:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.318614:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81969152 00000020:00010000:2.0:1789760797.318620:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.318624:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.318627:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.318629:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.318632:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.318639:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.318645:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.318659:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c900 00000008:00100000:2.0:1789760797.318736:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26680 - object 0x280000401:2 offset 81965056<>81969152 00000100:00100000:3.0:1789760797.318824:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760797.318839:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503951616:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.334309:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987318 gen 2 00000008:00010000:3.0:1789760797.334345:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c900 00000020:00200000:3.0:1789760797.334352:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.334362:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.334385:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c900 00000100:00100000:3.0:1789760797.334413:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503951616:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.334505:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20011, 20011] 00000080:00200000:2.0:1789760797.334517:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.334521:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.334534:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81969152 i_blocks 160120 00000020:00200000:2.0:1789760797.334537:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.334555:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.334582:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81969152, count: 4096 00000080:00200000:2.0:1789760797.334598:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81969152, bytes: 4096 00000080:00200000:2.0:1789760797.334606:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.334612:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.334615:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81969152, 81973248) 0 0 00000080:00200000:2.0:1789760797.334619:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.334630:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20012, 20012] 00020000:00200000:2.0:1789760797.334637:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81969152 chunk: [81969152, 81973248) 81973248, 4096 00020000:00200000:2.0:1789760797.334640:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.334655:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20012, 20012] 00000008:00010000:2.0:1789760797.334668:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.334673:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.334681:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81969152, 81973248) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.334822:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81969152=0x4e2c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.334867:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.334870:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81973248 00000020:00010000:2.0:1789760797.334876:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.334879:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.334882:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.334884:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.334887:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.334893:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.334898:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.334911:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000008:00100000:2.0:1789760797.334985:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25500 - object 0x280000401:2 offset 81969152<>81973248 00000100:00100000:3.0:1789760797.335120:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503951744:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.351512:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987319 gen 2 00000008:00010000:3.0:1789760797.351558:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000020:00200000:3.0:1789760797.351565:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.351575:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.351598:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c3c0 00000100:00100000:3.0:1789760797.351622:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503951744:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.351876:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20012, 20012] 00000080:00200000:2.0:1789760797.351888:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.351894:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.351907:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81973248 i_blocks 160128 00000020:00200000:2.0:1789760797.351911:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.351918:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.351944:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81973248, count: 4096 00000080:00200000:2.0:1789760797.351949:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81973248, bytes: 4096 00000080:00200000:2.0:1789760797.351957:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.351965:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.351967:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81973248, 81977344) 0 0 00000080:00200000:2.0:1789760797.351971:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.351983:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20013, 20013] 00020000:00200000:2.0:1789760797.351991:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81973248 chunk: [81973248, 81977344) 81977344, 4096 00020000:00200000:2.0:1789760797.351995:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.352009:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20013, 20013] 00000008:00010000:2.0:1789760797.352023:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.352028:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.352073:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81973248, 81977344) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.352085:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81973248=0x4e2d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.352128:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.352131:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81977344 00000020:00010000:2.0:1789760797.352137:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.352140:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.352143:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.352146:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.352149:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.352156:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.352162:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.352176:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cb40 00000008:00100000:2.0:1789760797.352253:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24000 - object 0x280000401:2 offset 81973248<>81977344 00000100:00100000:3.0:1789760797.352368:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760797.352382:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503951872:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.367389:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987320 gen 2 00000008:00010000:3.0:1789760797.367427:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cb40 00000020:00200000:3.0:1789760797.367434:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.367442:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.367462:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cb40 00000100:00100000:3.0:1789760797.367486:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503951872:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.367581:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20013, 20013] 00000080:00200000:2.0:1789760797.367604:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.367610:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.367623:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81977344 i_blocks 160136 00000020:00200000:2.0:1789760797.367627:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.367635:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.367662:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81977344, count: 4096 00000080:00200000:2.0:1789760797.367668:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81977344, bytes: 4096 00000080:00200000:2.0:1789760797.367675:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.367683:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.367694:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81977344, 81981440) 0 0 00000080:00200000:2.0:1789760797.367708:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.367721:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20014, 20014] 00020000:00200000:2.0:1789760797.367727:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81977344 chunk: [81977344, 81981440) 81981440, 4096 00020000:00200000:2.0:1789760797.367731:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.367747:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20014, 20014] 00000008:00010000:2.0:1789760797.367762:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.367766:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.367775:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81977344, 81981440) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.367786:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81977344=0x4e2e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.367834:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.367837:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81981440 00000020:00010000:2.0:1789760797.367844:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.367847:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.367849:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.367852:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.367855:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.367862:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.367866:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.367880:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c600 00000008:00100000:2.0:1789760797.367958:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24000 - object 0x280000401:2 offset 81977344<>81981440 00000100:00100000:3.0:1789760797.368105:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503952000:192.168.203.158@tcp:4:dd.0 00000100:00100000:2.0:1789760797.376168:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503952128:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760797.377526:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760797.377543:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503952256:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760797.378450:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503952384:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760797.378595:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2392:1876699503952640:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760797.379323:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503952512:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760797.379853:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503952128:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760797.379918:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503952256:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760797.382290:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987321 gen 2 00000008:00010000:2.0:1789760797.382529:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:2.0:1789760797.382539:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.382559:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.382692:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:2.0:1789760797.382734:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503952000:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760797.383503:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20014, 20014] 00000080:00200000:3.0:1789760797.383521:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.383529:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.383553:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81981440 i_blocks 160144 00000020:00200000:3.0:1789760797.383559:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.383571:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.383945:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81981440, count: 4096 00000080:00200000:3.0:1789760797.383956:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81981440, bytes: 4096 00000080:00200000:3.0:1789760797.383966:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.383976:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760797.383979:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81981440, 81985536) 0 0 00000080:00200000:3.0:1789760797.383983:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.384002:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20015, 20015] 00020000:00200000:3.0:1789760797.384011:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81981440 chunk: [81981440, 81985536) 81985536, 4096 00020000:00200000:3.0:1789760797.384015:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.384671:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20015, 20015] 00000100:00100000:2.0:1789760797.385014:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503952384:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760797.385064:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2392:1876699503952640:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760797.385121:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503952512:192.168.203.158@tcp:400:kworker.0 00000008:00010000:3.0:1789760797.385209:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.385217:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307560, flags 0x0 00000080:00200000:2.0:1789760797.390865:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81981440, 81985536) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.390894:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81981440=0x4e2f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.390998:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.391002:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81985536 00000020:00010000:2.0:1789760797.391013:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.391017:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.391022:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.391026:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.391029:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.391060:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.391070:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.391094:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c600 00000008:00100000:2.0:1789760797.391225:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24000 - object 0x280000401:2 offset 81981440<>81985536 00000100:00100000:2.0:1789760797.391288:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760797.391315:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503952768:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.408355:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987322 gen 2 00000008:00010000:3.0:1789760797.408406:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c600 00000020:00200000:3.0:1789760797.408415:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.408427:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.408449:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c600 00000100:00100000:3.0:1789760797.408483:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503952768:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760797.408607:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20015, 20015] 00000080:00200000:0.0:1789760797.408624:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.408630:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.408645:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81985536 i_blocks 160152 00000020:00200000:0.0:1789760797.408650:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.408660:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.408697:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81985536, count: 4096 00000080:00200000:0.0:1789760797.408705:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81985536, bytes: 4096 00000080:00200000:0.0:1789760797.408715:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.408726:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760797.408728:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81985536, 81989632) 0 0 00000080:00200000:0.0:1789760797.408732:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.408760:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20016, 20016] 00020000:00200000:0.0:1789760797.408770:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81985536 chunk: [81985536, 81989632) 81989632, 4096 00020000:00200000:0.0:1789760797.408775:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.408795:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20016, 20016] 00000008:00010000:0.0:1789760797.408815:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.408821:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760797.408834:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81985536, 81989632) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.408849:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81985536=0x4e30000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.408910:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.408913:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81989632 00000020:00010000:0.0:1789760797.408922:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.408926:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.408929:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.408932:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.408935:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.408944:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.408951:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.408969:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c000 00000008:00100000:0.0:1789760797.409110:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99880 - object 0x280000401:2 offset 81985536<>81989632 00000100:00100000:1.0:1789760797.409360:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503952896:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760797.438522:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987323 gen 2 00000008:00010000:1.0:1789760797.438587:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c000 00000020:00200000:1.0:1789760797.438595:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.438609:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.438634:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c000 00000100:00100000:1.0:1789760797.438668:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503952896:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760797.438791:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20016, 20016] 00000080:00200000:0.0:1789760797.438809:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.438815:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.438833:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81989632 i_blocks 160160 00000020:00200000:0.0:1789760797.438837:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.438847:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.438882:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81989632, count: 4096 00000080:00200000:0.0:1789760797.438889:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81989632, bytes: 4096 00000080:00200000:0.0:1789760797.438914:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.438923:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760797.438926:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81989632, 81993728) 0 0 00000080:00200000:0.0:1789760797.438930:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.439110:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20017, 20017] 00020000:00200000:0.0:1789760797.439120:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81989632 chunk: [81989632, 81993728) 81993728, 4096 00020000:00200000:0.0:1789760797.439125:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.439146:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20017, 20017] 00000008:00010000:0.0:1789760797.439166:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.439172:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760797.439184:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81989632, 81993728) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.439199:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81989632=0x4e31000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.439262:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.439265:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81993728 00000020:00010000:0.0:1789760797.439273:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.439277:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.439280:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.439284:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.439288:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.439296:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.439302:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.439319:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135ccc0 00000008:00100000:0.0:1789760797.439426:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98e00 - object 0x280000401:2 offset 81989632<>81993728 00000100:00100000:1.0:1789760797.439501:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503953024:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760797.460440:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987324 gen 2 00000008:00010000:1.0:1789760797.460491:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135ccc0 00000020:00200000:1.0:1789760797.460500:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.460512:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.460539:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135ccc0 00000100:00100000:1.0:1789760797.460585:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503953024:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760797.460719:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20017, 20017] 00000080:00200000:0.0:1789760797.460745:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.460753:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.460770:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81993728 i_blocks 160168 00000020:00200000:0.0:1789760797.460774:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.460785:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.460821:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81993728, count: 4096 00000080:00200000:0.0:1789760797.460828:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81993728, bytes: 4096 00000080:00200000:0.0:1789760797.460838:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.460847:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760797.460849:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81993728, 81997824) 0 0 00000080:00200000:0.0:1789760797.460854:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.460870:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20018, 20018] 00020000:00200000:0.0:1789760797.460878:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81993728 chunk: [81993728, 81997824) 81997824, 4096 00020000:00200000:0.0:1789760797.460883:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.460904:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20018, 20018] 00000008:00010000:0.0:1789760797.460924:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.460930:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760797.460954:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81993728, 81997824) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.460969:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81993728=0x4e32000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.461026:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.461028:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 81997824 00000020:00010000:0.0:1789760797.461082:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.461086:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.461089:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.461092:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.461096:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.461104:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.461111:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.461128:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c0c0 00000008:00100000:0.0:1789760797.461240:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98a80 - object 0x280000401:2 offset 81993728<>81997824 00000100:00100000:1.0:1789760797.461455:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503953152:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760797.476757:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987325 gen 2 00000008:00010000:1.0:1789760797.476812:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c0c0 00000020:00200000:1.0:1789760797.476821:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.476834:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.476932:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c0c0 00000100:00100000:1.0:1789760797.476965:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503953152:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760797.477476:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20018, 20018] 00000080:00200000:0.0:1789760797.477493:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.477499:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.477516:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 81997824 i_blocks 160176 00000020:00200000:0.0:1789760797.477521:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.477531:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.477578:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 81997824, count: 4096 00000080:00200000:0.0:1789760797.477585:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 81997824, bytes: 4096 00000080:00200000:0.0:1789760797.477600:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.477610:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760797.477612:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [81997824, 82001920) 0 0 00000080:00200000:0.0:1789760797.477617:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.477634:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20019, 20019] 00020000:00200000:0.0:1789760797.477645:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 81997824 chunk: [81997824, 82001920) 82001920, 4096 00020000:00200000:0.0:1789760797.477649:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.477670:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20019, 20019] 00000008:00010000:0.0:1789760797.477689:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.477695:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760797.477707:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [81997824, 82001920) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.477721:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=81997824=0x4e33000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.477881:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.477884:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82001920 00000020:00010000:0.0:1789760797.477893:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.477897:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.477901:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.477904:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.477908:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.477917:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.478149:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.478175:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c9c0 00000008:00100000:0.0:1789760797.478291:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98000 - object 0x280000401:2 offset 81997824<>82001920 00000100:00100000:1.0:1789760797.478384:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503953280:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760797.491678:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987326 gen 2 00000008:00010000:1.0:1789760797.492458:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c9c0 00000020:00200000:1.0:1789760797.492468:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.492481:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.492582:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c9c0 00000100:00100000:1.0:1789760797.492617:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503953280:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760797.494355:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20019, 20019] 00000080:00200000:0.0:1789760797.494378:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.494385:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.494405:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82001920 i_blocks 160184 00000020:00200000:0.0:1789760797.494410:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.494425:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.494462:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82001920, count: 4096 00000080:00200000:0.0:1789760797.494470:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82001920, bytes: 4096 00000080:00200000:0.0:1789760797.494486:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.494494:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760797.494497:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82001920, 82006016) 0 0 00000080:00200000:0.0:1789760797.494501:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.494518:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20020, 20020] 00020000:00200000:0.0:1789760797.494527:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82001920 chunk: [82001920, 82006016) 82006016, 4096 00020000:00200000:0.0:1789760797.494531:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.494563:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20020, 20020] 00000008:00010000:0.0:1789760797.494582:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.494589:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760797.494600:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82001920, 82006016) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.494615:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82001920=0x4e34000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.494683:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.494686:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82006016 00000020:00010000:0.0:1789760797.494694:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.494698:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.494701:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.494704:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.494707:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.494716:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.494722:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.494751:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760797.494864:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9a680 - object 0x280000401:2 offset 82001920<>82006016 00000100:00100000:1.0:1789760797.494958:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503953408:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760797.520145:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987327 gen 2 00000008:00010000:1.0:1789760797.520197:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:1.0:1789760797.520204:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.520216:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.523676:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:1.0:1789760797.523726:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503953408:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.526975:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20020, 20020] 00000080:00200000:2.0:1789760797.527009:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.527017:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.527054:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82006016 i_blocks 160192 00000020:00200000:2.0:1789760797.527061:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.527079:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.527115:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82006016, count: 4096 00000080:00200000:2.0:1789760797.527123:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82006016, bytes: 4096 00000080:00200000:2.0:1789760797.527138:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.527147:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.527149:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82006016, 82010112) 0 0 00000080:00200000:2.0:1789760797.527154:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.527171:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20021, 20021] 00020000:00200000:2.0:1789760797.527180:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82006016 chunk: [82006016, 82010112) 82010112, 4096 00020000:00200000:2.0:1789760797.527184:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.527206:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20021, 20021] 00000008:00010000:2.0:1789760797.527224:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.527231:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.527244:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82006016, 82010112) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.527259:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82006016=0x4e35000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.527333:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.527336:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82010112 00000020:00010000:2.0:1789760797.527345:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.527348:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.527352:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.527355:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.527358:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.527368:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.527375:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.527664:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c540 00000008:00100000:2.0:1789760797.527778:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25500 - object 0x280000401:2 offset 82006016<>82010112 00000100:00100000:3.0:1789760797.527880:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760797.527909:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503953536:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.545015:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987328 gen 2 00000008:00010000:3.0:1789760797.545370:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760797.545379:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.545390:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.545412:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760797.545441:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503953536:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.545575:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20021, 20021] 00000080:00200000:2.0:1789760797.545851:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.545858:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.545999:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82010112 i_blocks 160200 00000020:00200000:2.0:1789760797.546004:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.546017:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.546083:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82010112, count: 4096 00000080:00200000:2.0:1789760797.546091:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82010112, bytes: 4096 00000080:00200000:2.0:1789760797.546105:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.546114:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.546116:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82010112, 82014208) 0 0 00000080:00200000:2.0:1789760797.546121:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.546138:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20022, 20022] 00020000:00200000:2.0:1789760797.546147:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82010112 chunk: [82010112, 82014208) 82014208, 4096 00020000:00200000:2.0:1789760797.546152:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.546170:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20022, 20022] 00000008:00010000:2.0:1789760797.546188:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.546194:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.546205:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82010112, 82014208) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.546219:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82010112=0x4e36000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.546273:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.546276:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82014208 00000020:00010000:2.0:1789760797.546284:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.546287:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.546290:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.546294:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.546297:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.546304:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.546310:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.546329:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000008:00100000:2.0:1789760797.546437:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26680 - object 0x280000401:2 offset 82010112<>82014208 00000100:00100000:3.0:1789760797.546518:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503953664:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.567614:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987329 gen 2 00000008:00010000:3.0:1789760797.567730:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760797.567748:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.567760:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.567781:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760797.567813:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503953664:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.567935:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20022, 20022] 00000080:00200000:2.0:1789760797.567952:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.567958:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.567974:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82014208 i_blocks 160208 00000020:00200000:2.0:1789760797.567979:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.567990:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.568025:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82014208, count: 4096 00000080:00200000:2.0:1789760797.568032:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82014208, bytes: 4096 00000080:00200000:2.0:1789760797.568067:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.568076:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.568079:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82014208, 82018304) 0 0 00000080:00200000:2.0:1789760797.568083:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.568100:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20023, 20023] 00020000:00200000:2.0:1789760797.568110:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82014208 chunk: [82014208, 82018304) 82018304, 4096 00020000:00200000:2.0:1789760797.568115:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.568133:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20023, 20023] 00000008:00010000:2.0:1789760797.568151:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.568158:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.568170:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82014208, 82018304) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.568184:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82014208=0x4e37000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.568242:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.568245:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82018304 00000020:00010000:2.0:1789760797.568254:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.568257:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.568260:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.568263:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.568267:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.568275:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.568282:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.568299:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c840 00000008:00100000:2.0:1789760797.568409:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27480 - object 0x280000401:2 offset 82014208<>82018304 00000100:00100000:3.0:1789760797.568741:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503953792:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.588933:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987330 gen 2 00000008:00010000:3.0:1789760797.588987:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760797.588997:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.589009:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.589056:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760797.589089:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503953792:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.589249:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20023, 20023] 00000080:00200000:2.0:1789760797.589266:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.589273:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.589289:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82018304 i_blocks 160216 00000020:00200000:2.0:1789760797.589293:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.589304:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.589347:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82018304, count: 4096 00000080:00200000:2.0:1789760797.589355:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82018304, bytes: 4096 00000080:00200000:2.0:1789760797.589365:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.589374:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.589376:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82018304, 82022400) 0 0 00000080:00200000:2.0:1789760797.589381:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.589397:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20024, 20024] 00020000:00200000:2.0:1789760797.589405:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82018304 chunk: [82018304, 82022400) 82022400, 4096 00020000:00200000:2.0:1789760797.589410:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.589430:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20024, 20024] 00000008:00010000:2.0:1789760797.589449:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.589455:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.589466:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82018304, 82022400) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.589480:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82018304=0x4e38000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.589560:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.589564:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82022400 00000020:00010000:2.0:1789760797.589573:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.589576:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.589579:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.589583:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.589586:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.589595:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.589602:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.589619:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cd80 00000008:00100000:2.0:1789760797.589730:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24e00 - object 0x280000401:2 offset 82018304<>82022400 00000100:00100000:3.0:1789760797.589908:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503953920:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.607236:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987331 gen 2 00000008:00010000:3.0:1789760797.607289:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760797.607299:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.607310:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.607333:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760797.607366:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503953920:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.607488:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20024, 20024] 00000080:00200000:2.0:1789760797.607515:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.607521:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.607932:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82022400 i_blocks 160224 00000020:00200000:2.0:1789760797.607938:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.607951:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.607988:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82022400, count: 4096 00000080:00200000:2.0:1789760797.607995:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82022400, bytes: 4096 00000080:00200000:2.0:1789760797.608008:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.608017:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.608019:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82022400, 82026496) 0 0 00000080:00200000:2.0:1789760797.608023:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.608120:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20025, 20025] 00020000:00200000:2.0:1789760797.608130:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82022400 chunk: [82022400, 82026496) 82026496, 4096 00020000:00200000:2.0:1789760797.608135:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.608159:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20025, 20025] 00000008:00010000:2.0:1789760797.608180:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.608186:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.608209:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82022400, 82026496) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.608224:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82022400=0x4e39000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.608297:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.608300:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82026496 00000020:00010000:2.0:1789760797.608309:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.608312:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.608315:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.608318:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.608321:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.608331:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.608337:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.608357:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c180 00000008:00100000:2.0:1789760797.608663:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26d80 - object 0x280000401:2 offset 82022400<>82026496 00000100:00100000:3.0:1789760797.608845:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503954048:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.628569:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987332 gen 2 00000008:00010000:3.0:1789760797.628620:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760797.628629:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.628649:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.628669:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760797.628700:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503954048:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.628814:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20025, 20025] 00000080:00200000:2.0:1789760797.628831:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.628838:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.628854:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82026496 i_blocks 160232 00000020:00200000:2.0:1789760797.628858:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.628877:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.628913:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82026496, count: 4096 00000080:00200000:2.0:1789760797.628921:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82026496, bytes: 4096 00000080:00200000:2.0:1789760797.628931:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.628940:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.628943:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82026496, 82030592) 0 0 00000080:00200000:2.0:1789760797.628948:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.628965:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20026, 20026] 00020000:00200000:2.0:1789760797.628974:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82026496 chunk: [82026496, 82030592) 82030592, 4096 00020000:00200000:2.0:1789760797.628979:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.628997:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20026, 20026] 00000008:00010000:2.0:1789760797.629013:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.629019:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.629030:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82026496, 82030592) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.629087:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82026496=0x4e3a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.629141:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.629144:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82030592 00000020:00010000:2.0:1789760797.629152:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.629155:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.629158:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.629162:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.629165:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.629173:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.629179:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.629195:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c780 00000008:00100000:2.0:1789760797.629302:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24700 - object 0x280000401:2 offset 82026496<>82030592 00000100:00100000:3.0:1789760797.629564:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503954176:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.645880:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987333 gen 2 00000008:00010000:3.0:1789760797.645933:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760797.645942:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.645955:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.646031:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760797.646118:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503954176:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.648684:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20026, 20026] 00000080:00200000:2.0:1789760797.649095:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.649102:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.649116:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82030592 i_blocks 160240 00000020:00200000:2.0:1789760797.649122:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.649135:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.649161:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82030592, count: 4096 00000080:00200000:2.0:1789760797.649168:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82030592, bytes: 4096 00000080:00200000:2.0:1789760797.649179:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.649186:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.649188:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82030592, 82034688) 0 0 00000080:00200000:2.0:1789760797.649191:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.649204:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20027, 20027] 00020000:00200000:2.0:1789760797.649222:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82030592 chunk: [82030592, 82034688) 82034688, 4096 00020000:00200000:2.0:1789760797.649228:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.649245:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20027, 20027] 00000008:00010000:2.0:1789760797.649261:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.649265:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.649274:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82030592, 82034688) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.649286:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82030592=0x4e3b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.649342:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.649346:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82034688 00000020:00010000:2.0:1789760797.649352:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.649355:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.649358:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.649361:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.649363:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.649371:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.649389:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.649412:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c780 00000008:00100000:2.0:1789760797.649509:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f24380 - object 0x280000401:2 offset 82030592<>82034688 00000100:00100000:3.0:1789760797.649826:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503954304:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.662160:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987334 gen 2 00000008:00010000:3.0:1789760797.662213:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c780 00000020:00200000:3.0:1789760797.662222:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.662234:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.662257:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c780 00000100:00100000:3.0:1789760797.662484:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503954304:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.662640:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20027, 20027] 00000080:00200000:2.0:1789760797.662657:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.662664:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.662680:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82034688 i_blocks 160248 00000020:00200000:2.0:1789760797.662683:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.662694:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.662731:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82034688, count: 4096 00000080:00200000:2.0:1789760797.662737:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82034688, bytes: 4096 00000080:00200000:2.0:1789760797.662748:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.662756:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.662759:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82034688, 82038784) 0 0 00000080:00200000:2.0:1789760797.662764:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.662781:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20028, 20028] 00020000:00200000:2.0:1789760797.662790:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82034688 chunk: [82034688, 82038784) 82038784, 4096 00020000:00200000:2.0:1789760797.662795:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.662815:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20028, 20028] 00000008:00010000:2.0:1789760797.662833:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.662840:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.662852:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82034688, 82038784) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.662866:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82034688=0x4e3c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.662923:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.662926:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82038784 00000020:00010000:2.0:1789760797.662934:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.662938:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.662942:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.662945:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.662948:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.662958:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.662965:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.662981:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c180 00000008:00100000:2.0:1789760797.663130:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25c00 - object 0x280000401:2 offset 82034688<>82038784 00000100:00100000:3.0:1789760797.664086:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503954432:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.679634:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987335 gen 2 00000008:00010000:3.0:1789760797.679741:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c180 00000020:00200000:3.0:1789760797.679750:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.679763:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.679786:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c180 00000100:00100000:3.0:1789760797.679821:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503954432:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.680267:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20028, 20028] 00000080:00200000:2.0:1789760797.680291:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.680298:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.680316:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82038784 i_blocks 160256 00000020:00200000:2.0:1789760797.680320:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.680332:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.680373:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82038784, count: 4096 00000080:00200000:2.0:1789760797.680381:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82038784, bytes: 4096 00000080:00200000:2.0:1789760797.680393:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.680402:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.680404:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82038784, 82042880) 0 0 00000080:00200000:2.0:1789760797.680409:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.680427:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20029, 20029] 00020000:00200000:2.0:1789760797.680437:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82038784 chunk: [82038784, 82042880) 82042880, 4096 00020000:00200000:2.0:1789760797.680442:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.680462:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20029, 20029] 00000008:00010000:2.0:1789760797.680481:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.680488:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.680501:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82038784, 82042880) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.680518:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82038784=0x4e3d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.680645:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.680648:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82042880 00000020:00010000:2.0:1789760797.680657:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.680660:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.680664:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.680668:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.680672:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.680682:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.680689:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.680708:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0cd80 00000008:00100000:2.0:1789760797.680825:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25180 - object 0x280000401:2 offset 82038784<>82042880 00000100:00100000:3.0:1789760797.681891:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503954560:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.710988:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987336 gen 2 00000008:00010000:3.0:1789760797.711143:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0cd80 00000020:00200000:3.0:1789760797.711151:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.711162:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.711182:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0cd80 00000100:00100000:3.0:1789760797.711210:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503954560:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.711211:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20029, 20029] 00000080:00200000:2.0:1789760797.711229:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.711234:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.711250:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82042880 i_blocks 160264 00000020:00200000:2.0:1789760797.711253:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.711263:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.711299:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82042880, count: 4096 00000080:00200000:2.0:1789760797.711306:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82042880, bytes: 4096 00000080:00200000:2.0:1789760797.711316:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.711325:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.711327:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82042880, 82046976) 0 0 00000080:00200000:2.0:1789760797.711332:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.711349:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20030, 20030] 00020000:00200000:2.0:1789760797.711358:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82042880 chunk: [82042880, 82046976) 82046976, 4096 00020000:00200000:2.0:1789760797.711363:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.711383:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20030, 20030] 00000008:00010000:2.0:1789760797.711401:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.711407:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.711420:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82042880, 82046976) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.711441:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82042880=0x4e3e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.711504:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.711507:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82046976 00000020:00010000:2.0:1789760797.711515:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.711519:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.711522:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.711526:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.711529:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.711537:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.711553:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.711569:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c840 00000008:00100000:2.0:1789760797.711686:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f27800 - object 0x280000401:2 offset 82042880<>82046976 00000100:00100000:3.0:1789760797.712136:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503954688:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.736190:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987337 gen 2 00000008:00010000:3.0:1789760797.736244:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c840 00000020:00200000:3.0:1789760797.736253:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.736265:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.736385:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c840 00000100:00100000:3.0:1789760797.736420:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503954688:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.738264:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20030, 20030] 00000080:00200000:2.0:1789760797.738290:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.738297:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.738317:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82046976 i_blocks 160272 00000020:00200000:2.0:1789760797.738323:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.738339:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.738379:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82046976, count: 4096 00000080:00200000:2.0:1789760797.738387:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82046976, bytes: 4096 00000080:00200000:2.0:1789760797.738402:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.738412:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.738415:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82046976, 82051072) 0 0 00000080:00200000:2.0:1789760797.738420:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.738438:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20031, 20031] 00020000:00200000:2.0:1789760797.738452:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82046976 chunk: [82046976, 82051072) 82051072, 4096 00020000:00200000:2.0:1789760797.738840:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.738865:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20031, 20031] 00000008:00010000:2.0:1789760797.738885:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.738891:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.738905:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82046976, 82051072) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.738921:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82046976=0x4e3f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.738995:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.738998:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82051072 00000020:00010000:2.0:1789760797.739006:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.739010:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.739014:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.739017:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.739021:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.739031:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.739067:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.739098:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000008:00100000:2.0:1789760797.739216:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f26a00 - object 0x280000401:2 offset 82046976<>82051072 00000100:00100000:3.0:1789760797.740170:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503954816:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.755186:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987338 gen 2 00000008:00010000:3.0:1789760797.755238:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000020:00200000:3.0:1789760797.755247:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.755259:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.755345:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c9c0 00000100:00100000:3.0:1789760797.755377:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503954816:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.755478:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20031, 20031] 00000080:00200000:2.0:1789760797.755502:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.755509:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.755525:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82051072 i_blocks 160280 00000020:00200000:2.0:1789760797.755528:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.755538:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.755579:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82051072, count: 4096 00000080:00200000:2.0:1789760797.755751:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82051072, bytes: 4096 00000080:00200000:2.0:1789760797.755763:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.755772:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.755774:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82051072, 82055168) 0 0 00000080:00200000:2.0:1789760797.755779:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.755796:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20032, 20032] 00020000:00200000:2.0:1789760797.755804:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82051072 chunk: [82051072, 82055168) 82055168, 4096 00020000:00200000:2.0:1789760797.755809:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.755829:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20032, 20032] 00000008:00010000:2.0:1789760797.755849:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.755855:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.755866:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82051072, 82055168) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.755880:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82051072=0x4e40000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.755936:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.755938:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82055168 00000020:00010000:2.0:1789760797.755947:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.755950:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.755953:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.755957:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.755959:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.755969:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.755977:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.755995:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c540 00000008:00100000:2.0:1789760797.758243:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25f80 - object 0x280000401:2 offset 82051072<>82055168 00000100:00100000:3.0:1789760797.758418:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760797.758953:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503954944:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.772962:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987339 gen 2 00000008:00010000:2.0:1789760797.773542:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:2.0:1789760797.773560:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.773572:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.773708:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:2.0:1789760797.773746:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503954944:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760797.773938:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20032, 20032] 00000080:00200000:1.0:1789760797.773955:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.773962:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.773980:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82055168 i_blocks 160288 00000020:00200000:1.0:1789760797.773983:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.773994:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.774031:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82055168, count: 4096 00000080:00200000:1.0:1789760797.774065:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82055168, bytes: 4096 00000080:00200000:1.0:1789760797.774143:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.774152:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760797.774155:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82055168, 82059264) 0 0 00000080:00200000:1.0:1789760797.774159:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.774177:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20033, 20033] 00020000:00200000:1.0:1789760797.774186:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82055168 chunk: [82055168, 82059264) 82059264, 4096 00020000:00200000:1.0:1789760797.774191:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.774213:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20033, 20033] 00000008:00010000:1.0:1789760797.774233:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.774240:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760797.774252:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82055168, 82059264) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.774266:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82055168=0x4e41000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.774329:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.774332:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82059264 00000020:00010000:1.0:1789760797.774340:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.774344:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.774347:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.774350:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.774353:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.774361:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.774367:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.774384:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760797.775414:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 82055168<>82059264 00000100:00100000:1.0:1789760797.775476:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760797.775510:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503955072:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760797.796831:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987340 gen 2 00000008:00010000:1.0:1789760797.797185:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:1.0:1789760797.797195:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.797205:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.797232:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:1.0:1789760797.797263:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503955072:192.168.203.158@tcp:4:dd.0 00000080:00200000:2.0:1789760797.797378:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20033, 20033] 00000080:00200000:2.0:1789760797.797395:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.797401:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.797541:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82059264 i_blocks 160296 00000020:00200000:2.0:1789760797.797554:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.797563:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.797597:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82059264, count: 4096 00000080:00200000:2.0:1789760797.797604:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82059264, bytes: 4096 00000080:00200000:2.0:1789760797.797681:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.797690:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:2.0:1789760797.797692:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82059264, 82063360) 0 0 00000080:00200000:2.0:1789760797.797696:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.797712:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20034, 20034] 00020000:00200000:2.0:1789760797.797720:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82059264 chunk: [82059264, 82063360) 82063360, 4096 00020000:00200000:2.0:1789760797.797725:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.797745:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20034, 20034] 00000008:00010000:2.0:1789760797.797829:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.797836:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c337d330, osc lock ffff9018d23be980, flags 0x0 00000080:00200000:2.0:1789760797.797847:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82059264, 82063360) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.797861:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82059264=0x4e42000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.797923:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.797926:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82063360 00000020:00010000:2.0:1789760797.797934:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.797938:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.797940:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c337d330 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.797944:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.797947:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c337d330 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.797956:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.797963:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.797981:0:761:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c5f0c540 00000008:00100000:2.0:1789760797.798231:0:761:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f25f80 - object 0x280000401:2 offset 82059264<>82063360 00000100:00100000:2.0:1789760797.798267:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760797.798289:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503955200:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760797.818369:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987341 gen 2 00000008:00010000:3.0:1789760797.818422:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c5f0c540 00000020:00200000:3.0:1789760797.818431:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.818443:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.818472:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c5f0c540 00000100:00100000:3.0:1789760797.818507:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503955200:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760797.819092:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20034, 20034] 00000080:00200000:1.0:1789760797.819110:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.819116:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.819132:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82063360 i_blocks 160304 00000020:00200000:1.0:1789760797.819136:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.819145:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.819179:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82063360, count: 4096 00000080:00200000:1.0:1789760797.819185:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82063360, bytes: 4096 00000080:00200000:1.0:1789760797.819198:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.819206:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760797.819209:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82063360, 82067456) 0 0 00000080:00200000:1.0:1789760797.819213:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.819229:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20035, 20035] 00020000:00200000:1.0:1789760797.819236:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82063360 chunk: [82063360, 82067456) 82067456, 4096 00020000:00200000:1.0:1789760797.819241:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.819260:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20035, 20035] 00000008:00010000:1.0:1789760797.819278:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.819283:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760797.819292:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82063360, 82067456) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.819306:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82063360=0x4e43000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.819361:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.819364:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82067456 00000020:00010000:1.0:1789760797.819371:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.819375:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.819378:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.819381:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.819384:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.819392:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.819398:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.819415:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d589c0 00000008:00100000:1.0:1789760797.819531:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 82063360<>82067456 00000100:00100000:0.0:1789760797.820403:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760797.820433:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503955328:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760797.837909:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987342 gen 2 00000008:00010000:0.0:1789760797.838242:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d589c0 00000020:00200000:0.0:1789760797.838251:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.838264:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.838322:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d589c0 00000100:00100000:0.0:1789760797.838353:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503955328:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760797.838442:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20035, 20035] 00000080:00200000:1.0:1789760797.838459:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.838465:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.838480:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82067456 i_blocks 160312 00000020:00200000:1.0:1789760797.838483:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.838493:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.838527:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82067456, count: 4096 00000080:00200000:1.0:1789760797.838534:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82067456, bytes: 4096 00000080:00200000:1.0:1789760797.838555:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.838563:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760797.838566:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82067456, 82071552) 0 0 00000080:00200000:1.0:1789760797.838570:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.838704:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20036, 20036] 00020000:00200000:1.0:1789760797.838714:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82067456 chunk: [82067456, 82071552) 82071552, 4096 00020000:00200000:1.0:1789760797.838719:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.838739:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20036, 20036] 00000008:00010000:1.0:1789760797.838758:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.838764:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760797.838776:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82067456, 82071552) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.838790:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82067456=0x4e44000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.838849:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.838852:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82071552 00000020:00010000:1.0:1789760797.838860:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.838863:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.838866:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.838869:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.838872:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.838881:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.838887:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.838903:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d583c0 00000008:00100000:1.0:1789760797.839006:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 82067456<>82071552 00000100:00100000:0.0:1789760797.839078:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503955456:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760797.853888:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987343 gen 2 00000008:00010000:0.0:1789760797.853940:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d583c0 00000020:00200000:0.0:1789760797.853949:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.853960:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.853986:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d583c0 00000100:00100000:0.0:1789760797.854020:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503955456:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760797.854781:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20036, 20036] 00000080:00200000:1.0:1789760797.854800:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.854807:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.854823:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82071552 i_blocks 160320 00000020:00200000:1.0:1789760797.854827:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.854837:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.854988:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82071552, count: 4096 00000080:00200000:1.0:1789760797.854997:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82071552, bytes: 4096 00000080:00200000:1.0:1789760797.855011:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.855019:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760797.855022:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82071552, 82075648) 0 0 00000080:00200000:1.0:1789760797.855026:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.855151:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20037, 20037] 00020000:00200000:1.0:1789760797.855160:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82071552 chunk: [82071552, 82075648) 82075648, 4096 00020000:00200000:1.0:1789760797.855165:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.855186:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20037, 20037] 00000008:00010000:1.0:1789760797.855205:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.855211:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760797.855223:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82071552, 82075648) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.855237:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82071552=0x4e45000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.855375:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.855378:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82075648 00000020:00010000:1.0:1789760797.855388:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.855391:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.855394:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.855397:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.855400:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.855408:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.855414:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.855510:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58c00 00000008:00100000:1.0:1789760797.855732:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 82071552<>82075648 00000100:00100000:0.0:1789760797.855969:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503955584:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760797.865688:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987344 gen 2 00000008:00010000:0.0:1789760797.865746:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58c00 00000020:00200000:0.0:1789760797.865756:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.865773:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.865795:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58c00 00000100:00100000:0.0:1789760797.865833:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503955584:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760797.866444:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20037, 20037] 00000080:00200000:1.0:1789760797.866462:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.866468:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.866485:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82075648 i_blocks 160328 00000020:00200000:1.0:1789760797.866489:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.866500:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.866524:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82075648, count: 4096 00000080:00200000:1.0:1789760797.866530:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82075648, bytes: 4096 00000080:00200000:1.0:1789760797.866540:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.866557:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760797.866560:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82075648, 82079744) 0 0 00000080:00200000:1.0:1789760797.866563:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.866575:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20038, 20038] 00020000:00200000:1.0:1789760797.866581:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82075648 chunk: [82075648, 82079744) 82079744, 4096 00020000:00200000:1.0:1789760797.866585:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.866600:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20038, 20038] 00000008:00010000:1.0:1789760797.866612:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.866617:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760797.866625:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82075648, 82079744) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.866634:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82075648=0x4e46000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.866679:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.866681:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82079744 00000020:00010000:1.0:1789760797.866687:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.866690:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.866693:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.866696:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.866698:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.866705:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.866711:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.866728:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58540 00000008:00100000:1.0:1789760797.866813:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 82075648<>82079744 00000100:00100000:0.0:1789760797.866983:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760797.867013:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503955712:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760797.882804:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987345 gen 2 00000008:00010000:0.0:1789760797.882922:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58540 00000020:00200000:0.0:1789760797.882931:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.882943:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.882969:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58540 00000100:00100000:0.0:1789760797.883002:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503955712:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760797.883162:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20038, 20038] 00000080:00200000:1.0:1789760797.883177:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.883185:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.883210:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82079744 i_blocks 160336 00000020:00200000:1.0:1789760797.883215:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.883225:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.883665:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82079744, count: 4096 00000080:00200000:1.0:1789760797.883737:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82079744, bytes: 4096 00000080:00200000:1.0:1789760797.883752:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.883760:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760797.883762:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82079744, 82083840) 0 0 00000080:00200000:1.0:1789760797.883768:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.883784:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20039, 20039] 00020000:00200000:1.0:1789760797.883793:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82079744 chunk: [82079744, 82083840) 82083840, 4096 00020000:00200000:1.0:1789760797.883798:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.883819:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20039, 20039] 00000008:00010000:1.0:1789760797.883838:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.883843:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760797.884920:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82079744, 82083840) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.884946:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82079744=0x4e47000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.885030:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.885069:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82083840 00000020:00010000:1.0:1789760797.885078:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.885081:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.885085:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.885088:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.885091:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.885103:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.885112:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.885144:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58a80 00000008:00100000:1.0:1789760797.885282:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3f800 - object 0x280000401:2 offset 82079744<>82083840 00000100:00100000:0.0:1789760797.886090:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760797.886121:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503955840:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760797.903237:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987346 gen 2 00000008:00010000:0.0:1789760797.903368:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58a80 00000020:00200000:0.0:1789760797.903378:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.903388:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.903416:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58a80 00000100:00100000:0.0:1789760797.903449:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503955840:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760797.903592:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20039, 20039] 00000080:00200000:1.0:1789760797.903609:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.903615:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.903640:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82083840 i_blocks 160344 00000020:00200000:1.0:1789760797.903644:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.903653:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.903697:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82083840, count: 4096 00000080:00200000:1.0:1789760797.903705:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82083840, bytes: 4096 00000080:00200000:1.0:1789760797.903716:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.903725:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760797.903727:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82083840, 82087936) 0 0 00000080:00200000:1.0:1789760797.903732:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.903750:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20040, 20040] 00020000:00200000:1.0:1789760797.903758:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82083840 chunk: [82083840, 82087936) 82087936, 4096 00020000:00200000:1.0:1789760797.903763:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.903783:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20040, 20040] 00000008:00010000:1.0:1789760797.903800:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.903807:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760797.903818:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82083840, 82087936) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.903832:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82083840=0x4e48000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.903900:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.903903:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82087936 00000020:00010000:1.0:1789760797.903911:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.903915:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.903918:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.903921:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.903924:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.903931:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.903938:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.903954:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d586c0 00000008:00100000:1.0:1789760797.904112:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c000 - object 0x280000401:2 offset 82083840<>82087936 00000100:00100000:0.0:1789760797.904550:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760797.904725:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503955968:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760797.925331:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987347 gen 2 00000008:00010000:0.0:1789760797.925386:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d586c0 00000020:00200000:0.0:1789760797.925395:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.925408:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.925434:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d586c0 00000100:00100000:0.0:1789760797.925465:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503955968:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760797.925747:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20040, 20040] 00000080:00200000:1.0:1789760797.925764:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.925771:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.925787:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82087936 i_blocks 160352 00000020:00200000:1.0:1789760797.925791:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.925801:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.925839:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82087936, count: 4096 00000080:00200000:1.0:1789760797.925916:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82087936, bytes: 4096 00000080:00200000:1.0:1789760797.925929:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.925938:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760797.925941:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82087936, 82092032) 0 0 00000080:00200000:1.0:1789760797.925946:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.925963:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20041, 20041] 00020000:00200000:1.0:1789760797.925972:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82087936 chunk: [82087936, 82092032) 82092032, 4096 00020000:00200000:1.0:1789760797.925978:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.925998:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20041, 20041] 00000008:00010000:1.0:1789760797.926019:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.926025:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760797.926062:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82087936, 82092032) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.926076:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82087936=0x4e49000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.926209:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.926212:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82092032 00000020:00010000:1.0:1789760797.926222:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.926226:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.926229:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.926233:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.926236:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.926245:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.926251:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.926270:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58300 00000008:00100000:1.0:1789760797.926375:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 82087936<>82092032 00000100:00100000:0.0:1789760797.926481:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760797.926504:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503956096:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760797.946808:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987348 gen 2 00000008:00010000:0.0:1789760797.946862:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58300 00000020:00200000:0.0:1789760797.946871:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.946883:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.946907:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58300 00000100:00100000:0.0:1789760797.946938:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503956096:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760797.947056:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20041, 20041] 00000080:00200000:1.0:1789760797.947075:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.947081:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.947098:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82092032 i_blocks 160360 00000020:00200000:1.0:1789760797.947183:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.947198:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.947238:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82092032, count: 4096 00000080:00200000:1.0:1789760797.947246:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82092032, bytes: 4096 00000080:00200000:1.0:1789760797.947257:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.947265:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760797.947267:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82092032, 82096128) 0 0 00000080:00200000:1.0:1789760797.947272:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.947289:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20042, 20042] 00020000:00200000:1.0:1789760797.947298:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82092032 chunk: [82092032, 82096128) 82096128, 4096 00020000:00200000:1.0:1789760797.947304:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.947323:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20042, 20042] 00000008:00010000:1.0:1789760797.947340:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.947346:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760797.947365:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82092032, 82096128) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.947379:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82092032=0x4e4a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.947449:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.947452:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82096128 00000020:00010000:1.0:1789760797.947465:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.947469:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.947472:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.947476:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.947482:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.947490:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.947497:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.947520:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58000 00000008:00100000:1.0:1789760797.947637:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 82092032<>82096128 00000100:00100000:0.0:1789760797.947712:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503956224:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760797.965674:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987349 gen 2 00000008:00010000:0.0:1789760797.965727:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58000 00000020:00200000:0.0:1789760797.965737:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.965749:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.965775:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58000 00000100:00100000:0.0:1789760797.965806:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503956224:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760797.965927:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20042, 20042] 00000080:00200000:1.0:1789760797.965943:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.965950:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.965966:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82096128 i_blocks 160368 00000020:00200000:1.0:1789760797.965970:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.965979:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.966013:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82096128, count: 4096 00000080:00200000:1.0:1789760797.966019:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82096128, bytes: 4096 00000080:00200000:1.0:1789760797.966028:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.966067:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760797.966069:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82096128, 82100224) 0 0 00000080:00200000:1.0:1789760797.966084:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.966101:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20043, 20043] 00020000:00200000:1.0:1789760797.966112:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82096128 chunk: [82096128, 82100224) 82100224, 4096 00020000:00200000:1.0:1789760797.966116:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.966136:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20043, 20043] 00000008:00010000:1.0:1789760797.966154:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.966161:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760797.966172:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82096128, 82100224) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.966185:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82096128=0x4e4b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.966242:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.966245:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82100224 00000020:00010000:1.0:1789760797.966254:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.966257:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.966261:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.966264:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.966380:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.966391:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.966398:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.966416:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d580c0 00000008:00100000:1.0:1789760797.966520:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 82096128<>82100224 00000100:00100000:0.0:1789760797.966807:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503956352:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760797.982751:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987350 gen 2 00000008:00010000:0.0:1789760797.982803:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d580c0 00000020:00200000:0.0:1789760797.982812:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.982824:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.982848:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d580c0 00000100:00100000:0.0:1789760797.982880:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503956352:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760797.983175:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20043, 20043] 00000080:00200000:1.0:1789760797.983193:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.983199:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.983215:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82100224 i_blocks 160376 00000020:00200000:1.0:1789760797.983219:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.983229:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.983338:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82100224, count: 4096 00000080:00200000:1.0:1789760797.983346:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82100224, bytes: 4096 00000080:00200000:1.0:1789760797.983357:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.983365:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760797.983368:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82100224, 82104320) 0 0 00000080:00200000:1.0:1789760797.983372:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.983388:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20044, 20044] 00020000:00200000:1.0:1789760797.983397:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82100224 chunk: [82100224, 82104320) 82104320, 4096 00020000:00200000:1.0:1789760797.983401:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.983421:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20044, 20044] 00000008:00010000:1.0:1789760797.983439:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.983446:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760797.983457:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82100224, 82104320) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.983471:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82100224=0x4e4c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.983681:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.983685:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82104320 00000020:00010000:1.0:1789760797.983693:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.983697:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.983701:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.983704:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.983707:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.983714:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.983722:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.983740:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58780 00000008:00100000:1.0:1789760797.983844:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 82100224<>82104320 00000100:00100000:0.0:1789760797.983928:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503956480:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760798.002496:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987351 gen 2 00000008:00010000:0.0:1789760798.002556:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58780 00000020:00200000:0.0:1789760798.002565:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.002578:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.003331:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58780 00000100:00100000:0.0:1789760798.003379:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503956480:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760798.010524:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20044, 20044] 00000080:00200000:3.0:1789760798.010565:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760798.010573:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760798.011486:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82104320 i_blocks 160384 00000020:00200000:3.0:1789760798.011498:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760798.011518:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760798.011560:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82104320, count: 4096 00000080:00200000:3.0:1789760798.011569:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82104320, bytes: 4096 00000080:00200000:3.0:1789760798.011653:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760798.011663:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760798.011665:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82104320, 82108416) 0 0 00000080:00200000:3.0:1789760798.011670:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760798.011688:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20045, 20045] 00020000:00200000:3.0:1789760798.011698:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82104320 chunk: [82104320, 82108416) 82108416, 4096 00020000:00200000:3.0:1789760798.011703:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760798.011728:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20045, 20045] 00000008:00010000:3.0:1789760798.011749:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760798.011755:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307300, flags 0x0 00000080:00200000:3.0:1789760798.011768:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82104320, 82108416) append 0 append lockpos 0 00000080:00200000:3.0:1789760798.011786:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82104320=0x4e4d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760798.011854:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760798.011857:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82108416 00000020:00010000:3.0:1789760798.011865:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760798.011869:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760798.011872:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760798.011876:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760798.011879:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760798.011887:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760798.011894:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760798.011919:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760798.012212:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070d880 - object 0x280000401:2 offset 82104320<>82108416 00000100:00100000:2.0:1789760798.012532:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503956608:192.168.203.158@tcp:4:dd.0 00010000:00010000:2.0:1789760798.014743:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760798.030109:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987352 gen 2 00000008:00010000:3.0:1789760798.030161:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760798.030169:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760798.030180:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760798.030206:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760798.030237:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503956608:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760798.030904:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20045, 20045] 00000080:00200000:0.0:1789760798.030923:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760798.030930:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.030947:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82108416 i_blocks 160392 00000020:00200000:0.0:1789760798.030951:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.030965:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.031003:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82108416, count: 4096 00000080:00200000:0.0:1789760798.031011:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82108416, bytes: 4096 00000080:00200000:0.0:1789760798.031023:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.031078:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760798.031081:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82108416, 82112512) 0 0 00000080:00200000:0.0:1789760798.031086:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.031105:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20046, 20046] 00020000:00200000:0.0:1789760798.031114:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82108416 chunk: [82108416, 82112512) 82112512, 4096 00020000:00200000:0.0:1789760798.031119:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.031139:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20046, 20046] 00000008:00010000:0.0:1789760798.031158:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.031165:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a289b0, osc lock ffff9018d0544390, flags 0x0 00000080:00200000:0.0:1789760798.031176:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82108416, 82112512) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.031189:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82108416=0x4e4e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760798.031327:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760798.031330:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82112512 00000020:00010000:1.0:1789760798.031340:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760798.031344:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.031348:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a289b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.031352:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760798.031356:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a289b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760798.031366:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760798.031374:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760798.031391:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760798.031520:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 82108416<>82112512 00000100:00100000:0.0:1789760798.031768:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503956736:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760798.045247:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987353 gen 2 00000008:00010000:0.0:1789760798.045297:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58240 00000020:00200000:0.0:1789760798.045318:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.045331:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.045353:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58240 00000100:00100000:0.0:1789760798.045662:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503956736:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760798.045724:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20046, 20046] 00000080:00200000:1.0:1789760798.045740:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760798.045746:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760798.045764:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82112512 i_blocks 160400 00000020:00200000:1.0:1789760798.045767:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760798.045778:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760798.045868:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82112512, count: 4096 00000080:00200000:1.0:1789760798.045876:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82112512, bytes: 4096 00000080:00200000:1.0:1789760798.045888:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760798.045896:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760798.045899:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82112512, 82116608) 0 0 00000080:00200000:1.0:1789760798.045903:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760798.045920:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20047, 20047] 00020000:00200000:1.0:1789760798.045929:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82112512 chunk: [82112512, 82116608) 82116608, 4096 00020000:00200000:1.0:1789760798.045934:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760798.045952:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20047, 20047] 00000008:00010000:1.0:1789760798.045970:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760798.045976:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760798.045987:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82112512, 82116608) append 0 append lockpos 0 00000080:00200000:1.0:1789760798.046000:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82112512=0x4e4f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760798.046963:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760798.046970:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82116608 00000020:00010000:1.0:1789760798.046982:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760798.046986:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.046991:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.046994:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760798.046998:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760798.047009:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760798.047018:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760798.047683:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760798.047805:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 82112512<>82116608 00000100:00100000:1.0:1789760798.048218:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760798.048246:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503956864:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760798.064123:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987354 gen 2 00000008:00010000:1.0:1789760798.064174:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:1.0:1789760798.064184:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760798.064195:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760798.064216:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:1.0:1789760798.064251:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503956864:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760798.064261:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20047, 20047] 00000080:00200000:0.0:1789760798.064282:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760798.064288:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.064306:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82116608 i_blocks 160408 00000020:00200000:0.0:1789760798.064310:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.064320:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.064353:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82116608, count: 4096 00000080:00200000:0.0:1789760798.064361:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82116608, bytes: 4096 00000080:00200000:0.0:1789760798.064371:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.064380:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760798.064382:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82116608, 82120704) 0 0 00000080:00200000:0.0:1789760798.064386:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.064403:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20048, 20048] 00020000:00200000:0.0:1789760798.064411:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82116608 chunk: [82116608, 82120704) 82120704, 4096 00020000:00200000:0.0:1789760798.064415:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.064433:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20048, 20048] 00000008:00010000:0.0:1789760798.064451:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.064456:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a28ab0, osc lock ffff9018d05451d0, flags 0x0 00000080:00200000:0.0:1789760798.064465:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82116608, 82120704) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.064478:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82116608=0x4e50000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760798.064530:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760798.064532:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82120704 00000020:00010000:0.0:1789760798.064540:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760798.064555:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.064558:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a28ab0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.064562:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760798.064564:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a28ab0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760798.064572:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760798.064578:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760798.064605:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760798.064701:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b480 - object 0x280000401:2 offset 82116608<>82120704 00000100:00100000:0.0:1789760798.064741:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760798.064756:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503956992:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760798.079444:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987355 gen 2 00000008:00010000:0.0:1789760798.079583:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:0.0:1789760798.079592:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.079605:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.079626:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:0.0:1789760798.079667:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503956992:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760798.079832:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20048, 20048] 00000080:00200000:1.0:1789760798.084441:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760798.084458:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760798.084480:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82120704 i_blocks 160416 00000020:00200000:1.0:1789760798.084486:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760798.084507:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760798.084556:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82120704, count: 4096 00000080:00200000:1.0:1789760798.084566:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82120704, bytes: 4096 00000080:00200000:1.0:1789760798.084582:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760798.084592:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760798.084595:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82120704, 82124800) 0 0 00000080:00200000:1.0:1789760798.084599:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760798.084617:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20049, 20049] 00020000:00200000:1.0:1789760798.084626:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82120704 chunk: [82120704, 82124800) 82124800, 4096 00020000:00200000:1.0:1789760798.084632:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760798.084654:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20049, 20049] 00000008:00010000:1.0:1789760798.084675:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760798.084683:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760798.084695:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82120704, 82124800) append 0 append lockpos 0 00000080:00200000:1.0:1789760798.084723:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82120704=0x4e51000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760798.084800:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760798.084803:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82124800 00000020:00010000:1.0:1789760798.084813:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760798.084817:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.084820:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.084823:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760798.084826:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760798.084835:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760798.084843:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760798.084873:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760798.084989:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 82120704<>82124800 00000100:00100000:1.0:1789760798.085066:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760798.085092:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503957120:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760798.103179:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987356 gen 2 00000008:00010000:0.0:1789760798.106837:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760798.106847:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.106857:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.106868:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760798.106889:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503957120:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760798.107831:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20049, 20049] 00000080:00200000:1.0:1789760798.107845:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760798.107851:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760798.107873:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82124800 i_blocks 160424 00000020:00200000:1.0:1789760798.107877:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760798.107887:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760798.107912:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82124800, count: 4096 00000080:00200000:1.0:1789760798.107917:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82124800, bytes: 4096 00000080:00200000:1.0:1789760798.107928:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760798.107934:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760798.107936:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82124800, 82128896) 0 0 00000080:00200000:1.0:1789760798.107940:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760798.107952:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20050, 20050] 00020000:00200000:1.0:1789760798.107959:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82124800 chunk: [82124800, 82128896) 82128896, 4096 00020000:00200000:1.0:1789760798.107963:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760798.107978:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20050, 20050] 00000008:00010000:1.0:1789760798.107993:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760798.107998:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760798.108007:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82124800, 82128896) append 0 append lockpos 0 00000080:00200000:1.0:1789760798.108018:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82124800=0x4e52000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760798.108109:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760798.108112:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82128896 00000020:00010000:1.0:1789760798.108119:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760798.108122:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.108125:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.108128:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760798.108130:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760798.108137:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760798.108141:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760798.108157:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760798.108238:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ce00 - object 0x280000401:2 offset 82124800<>82128896 00000100:00100000:0.0:1789760798.108414:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760798.108433:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503957248:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760798.121392:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987357 gen 2 00000008:00010000:0.0:1789760798.121427:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760798.121434:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.121442:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.121464:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:0.0:1789760798.121487:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503957248:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760798.121709:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20050, 20050] 00000080:00200000:1.0:1789760798.121722:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760798.121727:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760798.121738:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82128896 i_blocks 160432 00000020:00200000:1.0:1789760798.121741:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760798.121749:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760798.121774:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82128896, count: 4096 00000080:00200000:1.0:1789760798.121780:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82128896, bytes: 4096 00000080:00200000:1.0:1789760798.121788:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760798.121796:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760798.121798:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82128896, 82132992) 0 0 00000080:00200000:1.0:1789760798.121802:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760798.121816:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20051, 20051] 00020000:00200000:1.0:1789760798.121822:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82128896 chunk: [82128896, 82132992) 82132992, 4096 00020000:00200000:1.0:1789760798.121827:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760798.121841:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20051, 20051] 00000008:00010000:1.0:1789760798.121855:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760798.122182:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760798.122194:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82128896, 82132992) append 0 append lockpos 0 00000080:00200000:1.0:1789760798.122205:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82128896=0x4e53000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760798.122253:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760798.122255:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82132992 00000020:00010000:1.0:1789760798.122262:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760798.122265:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.122268:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.122271:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760798.122273:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760798.122279:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760798.122285:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760798.122298:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760798.122376:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3dc00 - object 0x280000401:2 offset 82128896<>82132992 00000100:00100000:0.0:1789760798.123401:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503957376:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760798.142537:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987358 gen 2 00000008:00010000:0.0:1789760798.142937:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760798.142944:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.142955:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.143124:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000100:00100000:0.0:1789760798.143152:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503957376:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760798.143227:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20051, 20051] 00000080:00200000:1.0:1789760798.143240:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760798.143246:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760798.143258:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82132992 i_blocks 160440 00000020:00200000:1.0:1789760798.143263:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760798.143271:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760798.143827:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82132992, count: 4096 00000080:00200000:1.0:1789760798.143836:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82132992, bytes: 4096 00000080:00200000:1.0:1789760798.143849:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760798.143856:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760798.143922:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82132992, 82137088) 0 0 00000080:00200000:1.0:1789760798.143926:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760798.143941:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20052, 20052] 00020000:00200000:1.0:1789760798.143948:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82132992 chunk: [82132992, 82137088) 82137088, 4096 00020000:00200000:1.0:1789760798.143952:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760798.143970:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20052, 20052] 00000008:00010000:1.0:1789760798.143985:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760798.143990:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760798.144154:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82132992, 82137088) append 0 append lockpos 0 00000080:00200000:1.0:1789760798.144165:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82132992=0x4e54000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760798.144218:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760798.144221:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82137088 00000020:00010000:1.0:1789760798.144228:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760798.144231:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.144234:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.144237:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760798.144240:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760798.144247:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760798.144253:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760798.144349:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58f00 00000008:00100000:1.0:1789760798.144502:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e300 - object 0x280000401:2 offset 82132992<>82137088 00000100:00100000:0.0:1789760798.144575:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503957504:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760798.164222:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987359 gen 2 00000008:00010000:0.0:1789760798.164260:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58f00 00000020:00200000:0.0:1789760798.164268:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.165985:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.166007:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58f00 00000100:00100000:0.0:1789760798.166061:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503957504:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760798.166219:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20052, 20052] 00000080:00200000:3.0:1789760798.166232:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760798.166237:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760798.166249:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82137088 i_blocks 160448 00000020:00200000:3.0:1789760798.166252:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760798.166260:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760798.166284:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82137088, count: 4096 00000080:00200000:3.0:1789760798.166290:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82137088, bytes: 4096 00000080:00200000:3.0:1789760798.166300:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760798.166307:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760798.166308:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82137088, 82141184) 0 0 00000080:00200000:3.0:1789760798.166312:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760798.166322:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20053, 20053] 00020000:00200000:3.0:1789760798.166329:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82137088 chunk: [82137088, 82141184) 82141184, 4096 00020000:00200000:3.0:1789760798.166332:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760798.166346:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20053, 20053] 00000008:00010000:3.0:1789760798.166358:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760798.166363:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307300, flags 0x0 00000080:00200000:3.0:1789760798.166371:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82137088, 82141184) append 0 append lockpos 0 00000080:00200000:3.0:1789760798.166381:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82137088=0x4e55000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760798.166425:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760798.166428:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82141184 00000020:00010000:3.0:1789760798.166434:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760798.166437:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760798.166440:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760798.166443:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760798.166445:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760798.166451:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760798.166456:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760798.166469:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760798.166562:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070d880 - object 0x280000401:2 offset 82137088<>82141184 00000100:00100000:2.0:1789760798.166642:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503957632:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760798.186596:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987360 gen 2 00000008:00010000:3.0:1789760798.186632:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760798.186638:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760798.186647:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760798.186673:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760798.186696:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503957632:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760798.186793:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20053, 20053] 00000080:00200000:0.0:1789760798.186805:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760798.186810:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.186824:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82141184 i_blocks 160456 00000020:00200000:0.0:1789760798.186828:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.186847:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.186874:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82141184, count: 4096 00000080:00200000:0.0:1789760798.186880:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82141184, bytes: 4096 00000080:00200000:0.0:1789760798.186887:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.186894:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760798.186896:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82141184, 82145280) 0 0 00000080:00200000:0.0:1789760798.186900:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.186913:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20054, 20054] 00020000:00200000:0.0:1789760798.186919:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82141184 chunk: [82141184, 82145280) 82145280, 4096 00020000:00200000:0.0:1789760798.186923:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.186938:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20054, 20054] 00000008:00010000:0.0:1789760798.186951:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.186956:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a28ab0, osc lock ffff9018d05451d0, flags 0x0 00000080:00200000:0.0:1789760798.186964:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82141184, 82145280) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.186975:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82141184=0x4e56000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760798.187018:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760798.187021:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82145280 00000020:00010000:0.0:1789760798.187028:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760798.187031:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.187076:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a28ab0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.187080:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760798.187083:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a28ab0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760798.187091:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760798.187097:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760798.187111:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760798.187184:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b480 - object 0x280000401:2 offset 82141184<>82145280 00000100:00100000:0.0:1789760798.187210:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760798.187222:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503957760:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760798.223512:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987361 gen 2 00000008:00010000:0.0:1789760798.223556:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:0.0:1789760798.223562:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.223570:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.223594:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:0.0:1789760798.223616:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503957760:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760798.223737:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20054, 20054] 00000080:00200000:3.0:1789760798.223748:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760798.223753:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760798.223766:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82145280 i_blocks 160464 00000020:00200000:3.0:1789760798.223770:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760798.223777:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760798.223806:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82145280, count: 4096 00000080:00200000:3.0:1789760798.223812:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82145280, bytes: 4096 00000080:00200000:3.0:1789760798.223819:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760798.223827:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:3.0:1789760798.223829:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82145280, 82149376) 0 0 00000080:00200000:3.0:1789760798.223832:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760798.223844:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20055, 20055] 00020000:00200000:3.0:1789760798.223850:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82145280 chunk: [82145280, 82149376) 82149376, 4096 00020000:00200000:3.0:1789760798.223853:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760798.223866:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20055, 20055] 00000008:00010000:3.0:1789760798.223878:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760798.223882:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c863a1b0, osc lock ffff9018d2307300, flags 0x0 00000080:00200000:3.0:1789760798.223890:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82145280, 82149376) append 0 append lockpos 0 00000080:00200000:3.0:1789760798.223900:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82145280=0x4e57000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760798.223940:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760798.223943:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82149376 00000020:00010000:3.0:1789760798.223949:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760798.223952:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760798.223955:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c863a1b0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760798.223957:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760798.223960:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c863a1b0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760798.223965:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760798.223970:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760798.223980:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018c86f1000 00000008:00100000:3.0:1789760798.224080:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d070d880 - object 0x280000401:2 offset 82145280<>82149376 00000100:00100000:3.0:1789760798.224112:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760798.224123:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503957888:192.168.203.158@tcp:4:dd.0 00000100:00100000:3.0:1789760798.246770:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987362 gen 2 00000008:00010000:3.0:1789760798.246825:0:2392:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018c86f1000 00000020:00200000:3.0:1789760798.246833:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760798.246845:0:2392:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760798.246876:0:2392:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018c86f1000 00000100:00100000:3.0:1789760798.246908:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503957888:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760798.247077:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20055, 20055] 00000080:00200000:0.0:1789760798.247094:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760798.247100:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.247118:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82149376 i_blocks 160472 00000020:00200000:0.0:1789760798.247122:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.247131:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.247166:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82149376, count: 4096 00000080:00200000:0.0:1789760798.247174:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82149376, bytes: 4096 00000080:00200000:0.0:1789760798.247186:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.247195:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760798.247198:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82149376, 82153472) 0 0 00000080:00200000:0.0:1789760798.247202:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.247219:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20056, 20056] 00020000:00200000:0.0:1789760798.247227:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82149376 chunk: [82149376, 82153472) 82153472, 4096 00020000:00200000:0.0:1789760798.247233:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.247252:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20056, 20056] 00000008:00010000:0.0:1789760798.247270:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.247276:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a28ab0, osc lock ffff9018d05451d0, flags 0x0 00000080:00200000:0.0:1789760798.247288:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82149376, 82153472) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.247303:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82149376=0x4e58000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760798.247370:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760798.247373:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82153472 00000020:00010000:0.0:1789760798.247381:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760798.247385:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.247388:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a28ab0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.247392:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760798.247395:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a28ab0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760798.247403:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760798.247410:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760798.247427:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760798.247771:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b480 - object 0x280000401:2 offset 82149376<>82153472 00000100:00100000:1.0:1789760798.247902:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503958016:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760798.262089:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987363 gen 2 00000008:00010000:1.0:1789760798.262145:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760798.262153:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760798.262165:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760798.262184:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760798.262214:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503958016:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760798.262320:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20056, 20056] 00000080:00200000:0.0:1789760798.262345:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760798.262351:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.262367:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82153472 i_blocks 160480 00000020:00200000:0.0:1789760798.262370:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.262379:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.262415:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82153472, count: 4096 00000080:00200000:0.0:1789760798.262422:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82153472, bytes: 4096 00000080:00200000:0.0:1789760798.262431:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.262451:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760798.262454:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82153472, 82157568) 0 0 00000080:00200000:0.0:1789760798.262458:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.262473:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20057, 20057] 00020000:00200000:0.0:1789760798.262481:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82153472 chunk: [82153472, 82157568) 82157568, 4096 00020000:00200000:0.0:1789760798.262486:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.262512:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20057, 20057] 00000008:00010000:0.0:1789760798.262530:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.262535:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a28ab0, osc lock ffff9018d05451d0, flags 0x0 00000080:00200000:0.0:1789760798.262554:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82153472, 82157568) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.262832:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82153472=0x4e59000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760798.262897:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760798.262900:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82157568 00000020:00010000:0.0:1789760798.262909:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760798.262913:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.262916:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a28ab0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.262919:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760798.262922:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a28ab0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760798.262932:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760798.262938:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760798.262957:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760798.263100:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b800 - object 0x280000401:2 offset 82153472<>82157568 00000100:00100000:1.0:1789760798.263304:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503958144:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760798.281092:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987364 gen 2 00000008:00010000:1.0:1789760798.281145:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760798.281155:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760798.281166:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760798.281188:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760798.281222:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503958144:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760798.281394:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20057, 20057] 00000080:00200000:0.0:1789760798.281410:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760798.281416:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.281433:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82157568 i_blocks 160488 00000020:00200000:0.0:1789760798.281446:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.281456:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.281491:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82157568, count: 4096 00000080:00200000:0.0:1789760798.281499:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82157568, bytes: 4096 00000080:00200000:0.0:1789760798.281513:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.281524:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760798.281526:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82157568, 82161664) 0 0 00000080:00200000:0.0:1789760798.281531:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.281558:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20058, 20058] 00020000:00200000:0.0:1789760798.281575:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82157568 chunk: [82157568, 82161664) 82161664, 4096 00020000:00200000:0.0:1789760798.281586:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.281607:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20058, 20058] 00000008:00010000:0.0:1789760798.281626:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.281632:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a28ab0, osc lock ffff9018d05451d0, flags 0x0 00000080:00200000:0.0:1789760798.281645:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82157568, 82161664) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.281659:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82157568=0x4e5a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760798.281715:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760798.281718:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82161664 00000020:00010000:0.0:1789760798.281726:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760798.281729:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.281732:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a28ab0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.281736:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760798.281738:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a28ab0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760798.281746:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760798.281753:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760798.281770:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760798.281888:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9aa00 - object 0x280000401:2 offset 82157568<>82161664 00000100:00100000:1.0:1789760798.282165:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503958272:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760798.306782:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987365 gen 2 00000008:00010000:1.0:1789760798.306835:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760798.306845:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760798.306857:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760798.306880:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760798.306911:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503958272:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760798.307502:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20058, 20058] 00000080:00200000:0.0:1789760798.307521:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760798.307527:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.307550:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82161664 i_blocks 160496 00000020:00200000:0.0:1789760798.307555:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.307566:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.307610:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82161664, count: 4096 00000080:00200000:0.0:1789760798.307617:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82161664, bytes: 4096 00000080:00200000:0.0:1789760798.307628:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.307637:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760798.307640:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82161664, 82165760) 0 0 00000080:00200000:0.0:1789760798.307644:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.307661:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20059, 20059] 00020000:00200000:0.0:1789760798.307670:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82161664 chunk: [82161664, 82165760) 82165760, 4096 00020000:00200000:0.0:1789760798.307675:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.307694:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20059, 20059] 00000008:00010000:0.0:1789760798.307713:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.307720:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a28ab0, osc lock ffff9018d05451d0, flags 0x0 00000080:00200000:0.0:1789760798.307731:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82161664, 82165760) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.307745:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82161664=0x4e5b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760798.307807:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760798.307820:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82165760 00000020:00010000:0.0:1789760798.307829:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760798.307833:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.307836:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a28ab0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.307839:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760798.307842:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a28ab0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760798.307851:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760798.307857:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760798.307875:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c780 00000008:00100000:0.0:1789760798.307975:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9b100 - object 0x280000401:2 offset 82161664<>82165760 00000100:00100000:1.0:1789760798.308142:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503958400:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760798.321861:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987366 gen 2 00000008:00010000:1.0:1789760798.321915:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c780 00000020:00200000:1.0:1789760798.321923:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760798.321935:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760798.321959:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c780 00000100:00100000:1.0:1789760798.321990:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503958400:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760798.322231:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20059, 20059] 00000080:00200000:0.0:1789760798.322248:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760798.322254:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.322270:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82165760 i_blocks 160504 00000020:00200000:0.0:1789760798.322274:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.322288:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.322324:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82165760, count: 4096 00000080:00200000:0.0:1789760798.322332:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82165760, bytes: 4096 00000080:00200000:0.0:1789760798.322343:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.322353:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760798.322355:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82165760, 82169856) 0 0 00000080:00200000:0.0:1789760798.322360:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.322377:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20060, 20060] 00020000:00200000:0.0:1789760798.322386:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82165760 chunk: [82165760, 82169856) 82169856, 4096 00020000:00200000:0.0:1789760798.322391:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.322411:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20060, 20060] 00000008:00010000:0.0:1789760798.322429:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.322435:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a28ab0, osc lock ffff9018d05451d0, flags 0x0 00000080:00200000:0.0:1789760798.322447:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82165760, 82169856) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.322461:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82165760=0x4e5c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760798.322522:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760798.322525:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82169856 00000020:00010000:0.0:1789760798.322534:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760798.323172:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.323175:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a28ab0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.323178:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760798.323180:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a28ab0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760798.323189:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760798.323196:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760798.323217:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c600 00000008:00100000:0.0:1789760798.323323:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b9ad80 - object 0x280000401:2 offset 82165760<>82169856 00000100:00100000:1.0:1789760798.323442:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503958528:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760798.342854:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987367 gen 2 00000008:00010000:1.0:1789760798.342909:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c600 00000020:00200000:1.0:1789760798.342918:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760798.342930:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760798.342955:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c600 00000100:00100000:1.0:1789760798.342988:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503958528:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760798.343177:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20060, 20060] 00000080:00200000:0.0:1789760798.343193:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760798.343200:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.343216:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82169856 i_blocks 160512 00000020:00200000:0.0:1789760798.343219:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.343229:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.343273:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82169856, count: 4096 00000080:00200000:0.0:1789760798.343280:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82169856, bytes: 4096 00000080:00200000:0.0:1789760798.343290:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.343301:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760798.343303:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82169856, 82173952) 0 0 00000080:00200000:0.0:1789760798.343307:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.343323:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20061, 20061] 00020000:00200000:0.0:1789760798.343332:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82169856 chunk: [82169856, 82173952) 82173952, 4096 00020000:00200000:0.0:1789760798.343336:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.343356:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20061, 20061] 00000008:00010000:0.0:1789760798.343373:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.343380:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a28ab0, osc lock ffff9018d05451d0, flags 0x0 00000080:00200000:0.0:1789760798.343391:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82169856, 82173952) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.343406:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82169856=0x4e5d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760798.343464:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760798.343467:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82173952 00000020:00010000:0.0:1789760798.343476:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760798.343479:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.343482:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a28ab0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.343486:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760798.343488:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a28ab0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760798.343698:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760798.343706:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760798.343787:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c3c0 00000008:00100000:0.0:1789760798.343900:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b98380 - object 0x280000401:2 offset 82169856<>82173952 00000100:00100000:1.0:1789760798.343965:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503958656:192.168.203.158@tcp:4:dd.0 00000100:00100000:1.0:1789760798.357704:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987368 gen 2 00000008:00010000:1.0:1789760798.357759:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c3c0 00000020:00200000:1.0:1789760798.357767:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760798.357780:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760798.357804:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c3c0 00000100:00100000:1.0:1789760798.357837:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503958656:192.168.203.158@tcp:4:dd.0 00000080:00200000:0.0:1789760798.358245:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20061, 20061] 00000080:00200000:0.0:1789760798.358270:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760798.358277:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.358293:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82173952 i_blocks 160520 00000020:00200000:0.0:1789760798.358297:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.358308:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.358343:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82173952, count: 4096 00000080:00200000:0.0:1789760798.358350:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82173952, bytes: 4096 00000080:00200000:0.0:1789760798.358361:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.358370:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760798.358372:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82173952, 82178048) 0 0 00000080:00200000:0.0:1789760798.358376:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.358393:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20062, 20062] 00020000:00200000:0.0:1789760798.358403:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82173952 chunk: [82173952, 82178048) 82178048, 4096 00020000:00200000:0.0:1789760798.358408:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.358429:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20062, 20062] 00000008:00010000:0.0:1789760798.358450:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.358457:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a28ab0, osc lock ffff9018d05451d0, flags 0x0 00000080:00200000:0.0:1789760798.358468:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82173952, 82178048) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.358482:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82173952=0x4e5e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760798.358869:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760798.358878:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82178048 00000020:00010000:0.0:1789760798.358889:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760798.358892:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.358895:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a28ab0 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.358899:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760798.358902:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a28ab0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760798.358910:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760798.358919:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760798.358941:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760798.359138:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99c00 - object 0x280000401:2 offset 82173952<>82178048 00000100:00100000:0.0:1789760798.359191:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503958784:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760798.372365:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987369 gen 2 00000008:00010000:0.0:1789760798.372419:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:0.0:1789760798.372428:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.372440:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.372461:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:0.0:1789760798.372502:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503958784:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760798.372861:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20062, 20062] 00000080:00200000:1.0:1789760798.372879:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760798.372885:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760798.372903:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82178048 i_blocks 160528 00000020:00200000:1.0:1789760798.372906:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760798.372919:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760798.372957:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82178048, count: 4096 00000080:00200000:1.0:1789760798.372965:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82178048, bytes: 4096 00000080:00200000:1.0:1789760798.372977:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760798.372987:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760798.372990:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82178048, 82182144) 0 0 00000080:00200000:1.0:1789760798.372995:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760798.373012:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20063, 20063] 00020000:00200000:1.0:1789760798.373022:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82178048 chunk: [82178048, 82182144) 82182144, 4096 00020000:00200000:1.0:1789760798.373027:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760798.373242:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20063, 20063] 00000008:00010000:1.0:1789760798.373262:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760798.373341:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760798.373356:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82178048, 82182144) append 0 append lockpos 0 00000080:00200000:1.0:1789760798.373371:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82178048=0x4e5f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760798.373434:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760798.373438:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82182144 00000020:00010000:1.0:1789760798.373445:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760798.373448:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.373451:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.373455:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760798.373458:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760798.373466:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760798.373473:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760798.373741:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58d80 00000008:00100000:1.0:1789760798.373853:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3ed80 - object 0x280000401:2 offset 82178048<>82182144 00000100:00100000:0.0:1789760798.373978:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503958912:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760798.387075:0:2390:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987370 gen 2 00000008:00010000:0.0:1789760798.387128:0:2390:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58d80 00000020:00200000:0.0:1789760798.387136:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.387146:0:2390:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.387165:0:2390:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58d80 00000100:00100000:0.0:1789760798.387189:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503958912:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760798.387355:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20063, 20063] 00000080:00200000:1.0:1789760798.387371:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760798.387378:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760798.387393:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82182144 i_blocks 160536 00000020:00200000:1.0:1789760798.387396:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760798.387407:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760798.387439:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82182144, count: 4096 00000080:00200000:1.0:1789760798.387446:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82182144, bytes: 4096 00000080:00200000:1.0:1789760798.387456:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760798.387464:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760798.387467:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82182144, 82186240) 0 0 00000080:00200000:1.0:1789760798.387471:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760798.387487:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20064, 20064] 00020000:00200000:1.0:1789760798.387709:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82182144 chunk: [82182144, 82186240) 82186240, 4096 00020000:00200000:1.0:1789760798.387715:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760798.387735:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20064, 20064] 00000008:00010000:1.0:1789760798.387754:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760798.387761:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760798.387772:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82182144, 82186240) append 0 append lockpos 0 00000080:00200000:1.0:1789760798.387787:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82182144=0x4e60000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760798.387848:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760798.387852:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82186240 00000020:00010000:1.0:1789760798.387860:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760798.387863:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.387867:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668f368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.387870:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760798.387872:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760798.387880:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760798.387888:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760798.387907:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58b40 00000008:00100000:1.0:1789760798.388012:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3c700 - object 0x280000401:2 offset 82182144<>82186240 00000100:00100000:0.0:1789760798.388150:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503959040:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760798.404860:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987371 gen 2 00000008:00010000:0.0:1789760798.404911:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d3d58b40 00000020:00200000:0.0:1789760798.404919:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.404930:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.405057:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d3d58b40 00000100:00100000:0.0:1789760798.405086:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503959040:192.168.203.158@tcp:4:dd.0 00000080:00200000:3.0:1789760798.405192:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20064, 20064] 00000080:00200000:3.0:1789760798.405209:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760798.405216:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.405363:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82186240 i_blocks 160544 00000020:00200000:0.0:1789760798.405368:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.405378:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.405416:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82186240, count: 4096 00000080:00200000:0.0:1789760798.405423:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82186240, bytes: 4096 00000080:00200000:0.0:1789760798.405436:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.405446:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:0.0:1789760798.405448:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82186240, 82190336) 0 0 00000080:00200000:0.0:1789760798.405453:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.405471:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20065, 20065] 00020000:00200000:0.0:1789760798.405480:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82186240 chunk: [82186240, 82190336) 82190336, 4096 00020000:00200000:0.0:1789760798.405485:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.405634:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20065, 20065] 00000008:00010000:0.0:1789760798.405719:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.405725:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c5a28ab0, osc lock ffff9018d05451d0, flags 0x0 00000080:00200000:0.0:1789760798.405738:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82186240, 82190336) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.405753:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82186240=0x4e61000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760798.405812:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760798.405814:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82190336 00000020:00010000:0.0:1789760798.405823:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760798.405826:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.405829:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c5a28ab0 (ffff9018c668fc68/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.405833:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760798.405836:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c5a28ab0 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760798.405845:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760798.405851:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760798.405865:0:12398:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d135c540 00000008:00100000:0.0:1789760798.405963:0:12398:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018c8b99c00 - object 0x280000401:2 offset 82186240<>82190336 00000100:00100000:0.0:1789760798.406204:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760798.406223:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503959168:192.168.203.158@tcp:4:dd.0 00000100:00100000:0.0:1789760798.423366:0:2391:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9018c54e9800: committing for last_committed 4294987372 gen 2 00000008:00010000:0.0:1789760798.423419:0:2391:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9018d135c540 00000020:00200000:0.0:1789760798.423429:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.423441:0:2391:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.423466:0:2391:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9018d135c540 00000100:00100000:0.0:1789760798.423500:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503959168:192.168.203.158@tcp:4:dd.0 00000080:00200000:1.0:1789760798.423630:0:12577:0:(file.c:2205:ll_file_io_generic()) Range unlock [20065, 20065] 00000080:00200000:1.0:1789760798.423646:0:12577:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760798.423652:0:12577:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760798.423669:0:12577:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6:0x0] updating i_size 82190336 i_blocks 160552 00000020:00200000:1.0:1789760798.423673:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760798.423683:0:12577:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760798.423720:0:12577:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000402:0x6:0x0], ppos: 82190336, count: 4096 00000080:00200000:1.0:1789760798.423727:0:12577:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 82190336, bytes: 4096 00000080:00200000:1.0:1789760798.424184:0:12577:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760798.424195:0:12577:0:(cl_io.c:214:cl_io_rw_init()) header@00000000bcae4365[0x110, 2, [0x200000402:0x6:0x0] hash] 00000020:00200000:1.0:1789760798.424197:0:12577:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [82190336, 82194432) 0 0 00000080:00200000:1.0:1789760798.424202:0:12577:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760798.424220:0:12577:0:(file.c:2168:ll_file_io_generic()) Range lock [20066, 20066] 00020000:00200000:1.0:1789760798.424229:0:12577:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 82190336 chunk: [82190336, 82194432) 82194432, 4096 00020000:00200000:1.0:1789760798.424233:0:12577:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760798.424254:0:12577:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [20066, 20066] 00000008:00010000:1.0:1789760798.424273:0:12577:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760798.424279:0:12577:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9018c8631b30, osc lock ffff9018d181f7c0, flags 0x0 00000080:00200000:1.0:1789760798.424292:0:12577:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [82190336, 82194432) append 0 append lockpos 0 00000080:00200000:1.0:1789760798.424306:0:12577:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000402:0x6:0x0](ffff9018c4985b88), size=4096 (max 536870912), offset=82190336=0x4e62000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760798.424372:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760798.424376:0:12577:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 82194432 00000020:00010000:1.0:1789760798.424385:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760798.424388:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.424391:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9018c8631b30 (ffff9018c668fc68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760798.424395:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018d1418b78 (ffff9018c668f7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760798.424398:0:12577:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9018c8631b30 (ffff9018c668f7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760798.424406:0:12577:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760798.424413:0:12577:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760798.424433:0:254:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9018d3d58240 00000008:00100000:1.0:1789760798.424812:0:254:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9018d0f3e680 - object 0x280000401:2 offset 82190336<>82194432 00000100:00100000:1.0:1789760798.424894:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760798.424914:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503959296:192.168.203.158@tcp:4:dd.0 00010000:00010000:0.0:1789760799.039770:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760800.062223:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760801.086239:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760802.110179:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760802.494257:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503959424:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760802.494274:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503959552:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760802.494376:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503959680:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760802.494382:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503959808:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760802.494414:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2392:1876699503959936:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760802.494422:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503960064:192.168.203.158@tcp:400:kworker.0 00010000:00010000:3.0:1789760803.134203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760804.158267:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760805.182218:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760806.206140:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760807.230184:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760807.614218:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503960320:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760807.614235:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503960192:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760807.614316:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503960576:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760807.614347:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503960832:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760807.614400:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503960448:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760807.614441:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2392:1876699503960704:192.168.203.158@tcp:400:kworker.0 00010000:00010000:0.0:1789760808.254202:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760809.278210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760810.302187:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760811.326188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760812.350209:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760812.734249:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503960960:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760812.734304:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503961088:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760812.734355:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503961344:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760812.734476:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503961600:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760812.734555:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503961216:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760812.734604:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2392:1876699503961472:192.168.203.158@tcp:400:kworker.0 00010000:00010000:2.0:1789760813.374197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760814.398155:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1789760814.782216:0:2390:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760798/real 1789760798] req@ffff9018d0f3e680 x1876699503959296/t0(0) o4->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789760814 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1789760814.789568:0:2390:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1789760814.789579:0:2390:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff9018c54e9800: Connection to lustre-OST0000 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1789760814.795322:0:2390:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00100000:0.0:1789760814.795521:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2390:1876699503961728:192.168.203.158@tcp:400:kworker.0 00000100:00080000:1.0:1789760814.795527:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760814.795536:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760814.795540:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760814.795557:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 365 00000100:00080000:1.0:1789760814.795575:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760814.795589:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789760814.795614:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f3ed80 x1876699503962112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760814.795641:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503961984:192.168.203.158@tcp:400:kworker.0 00000100:00100000:0.0:1789760814.795695:0:2390:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2390:1876699503962368:192.168.203.158@tcp:400:kworker.0 00000100:00100000:1.0:1789760814.795707:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503961856:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760814.795747:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503962112:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789760814.795794:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2391:1876699503962240:192.168.203.158@tcp:400:kworker.0 00010000:00010000:3.0:1789760815.422217:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1789760815.806248:0:11323:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1789760815.806269:0:11323:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760815.806299:0:11323:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1789760816.446278:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760817.470252:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760818.494266:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1789760818.878168:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760802/real 1789760802] req@ffff9018d0f24a80 x1876699503959552/t0(0) o400->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1789760818.878183:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760802/real 1789760802] req@ffff9018d0f26a00 x1876699503959936/t0(0) o400->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760818.878256:0:2392:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d0f26a00 x1876699503959936/t0(0) o400->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760818.878283:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1789760818.878287:0:2392:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9018c5884000: Connection to lustre-MDT0000 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:2.0:1789760818.885556:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d0f24a80 x1876699503959552/t0(0) o400->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760818.885566:0:2393:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1789760818.885569:0:2393:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9018c54e9800: Connection to lustre-MDT0000 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1789760818.885574:0:2393:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:2.0:1789760818.885593:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760802/real 1789760802] req@ffff9018d0f25f80 x1876699503959808/t0(0) o400->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760818.885603:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d0f25f80 x1876699503959808/t0(0) o400->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760818.885609:0:2393:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9018c855c000 already not connected (conn 3, was 2): CONNECTING 00000100:00000400:2.0:1789760818.885613:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760802/real 1789760802] req@ffff9018d0f27800 x1876699503960064/t0(0) o400->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760818.885621:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d0f27800 x1876699503960064/t0(0) o400->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760818.885626:0:2393:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1789760818.885629:0:2393:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff9018c5884000: Connection to lustre-MDT0001 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1789760818.885631:0:2393:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00100000:2.0:1789760818.885649:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503959552:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760818.885655:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503959808:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760818.885661:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503960064:192.168.203.158@tcp:400:kworker.0 00000100:00080000:1.0:1789760818.885733:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760818.885741:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760818.885746:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760818.885756:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 225 00000100:00100000:0.0:1789760818.885762:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8b68caea-910d-4e7e-8105-d8da387b4d8a:2391:1876699503962496:192.168.203.158@tcp:400:kworker.0 00000100:00080000:1.0:1789760818.885779:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760818.885795:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760818.885853:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699503962624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760818.885920:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760818.885926:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1789760818.885928:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760818.885937:0:2391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503962752:192.168.203.158@tcp:400:kworker.0 00000100:00100000:0.0:1789760818.886002:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503962624:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789760818.896694:0:2392:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00080000:1.0:1789760818.896697:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760818.896704:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760818.896708:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760818.896718:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 288 00000100:00000400:3.0:1789760818.896725:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760802/real 1789760802] req@ffff9018d0f26300 x1876699503959424/t0(0) o400->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760818.896732:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760818.896747:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760818.896766:0:2392:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d0f26300 x1876699503959424/t0(0) o400->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760818.896770:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699503962880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760818.896794:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from FULL to DISCONN 00000100:02020000:3.0:1789760818.896800:0:2392:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail 00000100:00080000:1.0:1789760818.896808:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760818.896813:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760818.896815:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760818.896818:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 288 00000100:00080000:1.0:1789760818.896823:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760818.896829:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760818.896839:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699503963008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760818.896870:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503962880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760818.896960:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503963008:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789760818.900161:0:2392:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.203.158@tcp for MGC192.168.203.158@tcp not replayable, auto-deactivating 00000100:00080000:3.0:1789760818.900164:0:2392:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:3.0:1789760818.900169:0:2392:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9018d0f26300 x1876699503959424/t0(0) o400->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760818.900180:0:2392:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9018d070d880 x1876699503960192/t0(0) o400->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760823 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760818.900188:0:2392:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9018d070ce00 x1876699503960960/t0(0) o400->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760828 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760818.900196:0:2392:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9018d0f3c000 x1876699503961728/t0(0) o400->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760830 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760818.900207:0:2392:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9018d0f3ce00 x1876699503962496/t0(0) o400->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760834 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760818.900224:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 10000000:01000000:3.0:1789760818.900225:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1789760818.900230:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 00000100:00080000:1.0:1789760818.900232:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 10000000:01000000:3.0:1789760818.900232:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:3.0:1789760818.900237:0:2392:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1789760818.900239:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789760818.900244:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8b68caea-910d-4e7e-8105-d8da387b4d8a:2391:1876699503962496:192.168.203.158@tcp:400:kworker.0 00000100:00080000:1.0:1789760818.900244:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:3.0:1789760818.900253:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760802/real 1789760802] req@ffff9018d0f27100 x1876699503959680/t0(0) o400->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760818.900255:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760818.900263:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760818.900267:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760818.900274:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 225 00000100:00100000:3.0:1789760818.900283:0:2392:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d0f27100 x1876699503959680/t0(0) o400->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760818 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760818.900286:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789760818.900297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1789760818.900303:0:2392:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff9018c54e9800: Connection to lustre-MDT0001 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1789760818.900306:0:2392:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00100000:1.0:1789760818.900311:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699503963136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760818.900325:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2390:1876699503961728:192.168.203.158@tcp:400:kworker.0 00000100:00080000:1.0:1789760818.900330:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760818.900332:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503960960:192.168.203.158@tcp:400:kworker.0 00000100:00080000:1.0:1789760818.900336:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760818.900340:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760818.900349:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 225 00000100:00080000:1.0:1789760818.900356:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760818.900360:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503960192:192.168.203.158@tcp:400:kworker.0 00000008:00100000:1.0:1789760818.900362:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760818.900373:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2392:1876699503959936:192.168.203.158@tcp:400:kworker.0 00000100:00100000:1.0:1789760818.900377:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699503963264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760818.900402:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8b68caea-910d-4e7e-8105-d8da387b4d8a:2392:1876699503959424:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760818.900404:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503963136:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760818.900425:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503959680:192.168.203.158@tcp:400:kworker.0 00000100:00080000:1.0:1789760818.900428:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760818.900435:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760818.900439:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760818.900442:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760818.900451:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760818.900453:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760818.900456:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760818.900459:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1789760818.900512:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503963264:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760819.518189:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760819.902335:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760819.902342:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760819.902345:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760819.902348:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760819.902351:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760819.902354:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760819.902356:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760819.902360:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760819.902362:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760819.902365:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760819.902367:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760819.902369:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760820.542243:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760821.566201:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760822.590200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760823.614148:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760823.934151:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760823.934161:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760823.934165:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760823.934168:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760823.934171:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760823.934174:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760823.934177:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760823.934180:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760823.934184:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760823.934187:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760823.934190:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760823.934194:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:2.0:1789760823.998150:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760807/real 1789760807] req@ffff9018d070f800 x1876699503960448/t0(0) o400->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1789760823.998150:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760807/real 1789760807] req@ffff9018d070fb80 x1876699503960320/t0(0) o400->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760823.998212:0:2392:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d070f800 x1876699503960448/t0(0) o400->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760823.998230:0:2392:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c855a000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:2.0:1789760823.998238:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760807/real 1789760807] req@ffff9018d070d500 x1876699503960704/t0(0) o400->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760823.998266:0:2392:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d070d500 x1876699503960704/t0(0) o400->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760823.998275:0:2392:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c888a000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1789760823.998292:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503960448:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760823.998302:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2392:1876699503960704:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760824.008419:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d070fb80 x1876699503960320/t0(0) o400->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760824.008428:0:2393:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c855e800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1789760824.008435:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760807/real 1789760807] req@ffff9018d070c000 x1876699503960576/t0(0) o400->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760824.008445:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d070c000 x1876699503960576/t0(0) o400->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760824.008449:0:2393:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9018c855c000 already not connected (conn 3, was 2): CONNECTING 00000100:00000400:3.0:1789760824.008453:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760807/real 1789760807] req@ffff9018d070dc00 x1876699503960832/t0(0) o400->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760824.008460:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d070dc00 x1876699503960832/t0(0) o400->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760824.008464:0:2393:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c7fd9800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:3.0:1789760824.008474:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503960320:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760824.008478:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503960576:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760824.008481:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503960832:192.168.203.158@tcp:400:kworker.0 00010000:00010000:3.0:1789760824.638174:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760825.022145:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760814/real 1789760814] req@ffff9018d0f3ed80 x1876699503962112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760825.022181:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f3ed80 x1876699503962112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760825.022195:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760825.022211:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789760825.022214:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503962112:192.168.203.158@tcp:8:kworker.0 00000100:00080000:1.0:1789760825.022236:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760825.022241:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760825.022244:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760825.022246:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760825.022249:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760825.022251:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760825.022253:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760825.022255:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760825.022257:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760825.022262:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 746 00000100:00080000:1.0:1789760825.022272:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1789760825.022278:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760825.022288:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789760825.022312:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699503963392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760825.022332:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760825.022335:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760825.022337:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760825.022350:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1789760825.022370:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503963392:192.168.203.158@tcp:8:kworker.0 00010000:00010000:3.0:1789760825.662182:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760826.686292:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760827.710261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760828.094205:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760818/real 1789760818] req@ffff9018c66a5c00 x1876699503963264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1789760828.094231:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760812/real 1789760812] req@ffff9018d070e680 x1876699503961216/t0(0) o400->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094255:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699503963264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094293:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760818/real 1789760818] req@ffff9018c66a6d80 x1876699503963136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094305:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699503963136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094314:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760818/real 1789760818] req@ffff9018c66a5f80 x1876699503962880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094356:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699503962880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094379:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760818/real 1789760818] req@ffff9018c66a4000 x1876699503963008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094396:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699503963008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094407:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760818/real 1789760818] req@ffff9018c66a7b80 x1876699503962624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094419:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699503962624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760828.094434:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760828.094459:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760828.094464:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503963264:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789760828.094470:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760828.094477:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760828.094481:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503963136:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789760828.094489:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760828.094499:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760828.094506:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760828.094510:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760828.094516:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789760828.094519:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 750 00000100:00100000:3.0:1789760828.094529:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503962880:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789760828.094535:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1789760828.094539:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760828.094549:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789760828.094554:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760828.094570:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503963008:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789760828.094579:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760828.094595:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760828.094599:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503962624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789760828.094617:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699503963520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760828.094642:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760828.094646:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760828.094648:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760828.094654:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 750 00000100:00080000:1.0:1789760828.094659:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1789760828.094663:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760828.094669:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503963520:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789760828.094678:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760828.094695:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699503963648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760828.094705:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760828.094708:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760828.094709:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760828.094712:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 750 00000100:00080000:1.0:1789760828.094715:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1789760828.094718:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760828.094722:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760828.094732:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699503963776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760828.094739:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760828.094743:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760828.094745:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760828.094747:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760828.094748:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760828.094750:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 750 00000100:00080000:1.0:1789760828.094754:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1789760828.094757:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760828.094760:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760828.094768:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699503963904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760828.094774:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760828.094778:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760828.094780:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760828.094784:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 750 00000100:00080000:1.0:1789760828.094788:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1789760828.094800:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760828.094804:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503963648:192.168.203.158@tcp:38:kworker.0 00000008:00100000:1.0:1789760828.094806:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760828.094822:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699503964032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760828.094839:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760828.094843:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760828.094848:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503963776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760828.094868:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503963904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760828.094888:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503964032:192.168.203.158@tcp:38:kworker.0 00000100:00000400:3.0:1789760828.094919:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760812/real 1789760812] req@ffff9018d070c380 x1876699503961088/t0(0) o400->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094933:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d070c380 x1876699503961088/t0(0) o400->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760828.094946:0:2393:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c855e800 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:3.0:1789760828.094951:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760812/real 1789760812] req@ffff9018d070e300 x1876699503961344/t0(0) o400->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094961:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d070e300 x1876699503961344/t0(0) o400->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760828.094967:0:2393:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9018c855c000 already not connected (conn 4, was 2): CONNECTING 00000100:00000400:3.0:1789760828.094981:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760812/real 1789760812] req@ffff9018d070d180 x1876699503961600/t0(0) o400->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760828.094991:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d070d180 x1876699503961600/t0(0) o400->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760828.094997:0:2393:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c7fd9800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:3.0:1789760828.095003:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503961088:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760828.095008:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2393:1876699503961344:192.168.203.158@tcp:400:kworker.0 00000100:00100000:3.0:1789760828.095012:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2393:1876699503961600:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760828.112950:0:2392:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d070e680 x1876699503961216/t0(0) o400->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760828.112965:0:2392:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c855a000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:2.0:1789760828.113000:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760812/real 1789760812] req@ffff9018d070f100 x1876699503961472/t0(0) o400->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760828.113023:0:2392:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d070f100 x1876699503961472/t0(0) o400->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760828 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760828.113078:0:2392:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c888a000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:2.0:1789760828.113103:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2392:1876699503961216:192.168.203.158@tcp:400:kworker.0 00000100:00100000:2.0:1789760828.113114:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2392:1876699503961472:192.168.203.158@tcp:400:kworker.0 00010000:00010000:3.0:1789760828.734199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:1.0:1789760829.566155:0:2381:0:(socklnd_cb.c:2254:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.203.158@tcp (192.168.203.158:988) timed out; the network or node may be down. 00010000:00010000:0.0:1789760829.758259:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1789760830.142167:0:2391:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760814/real 1789760814] req@ffff9018d0f3f800 x1876699503961856/t0(0) o400->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1789760830.142167:0:2390:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760814/real 1789760814] req@ffff9018d0f3dc00 x1876699503962368/t0(0) o400->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760830.142246:0:2391:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d0f3f800 x1876699503961856/t0(0) o400->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760830.142272:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789760830.142273:0:2390:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d0f3dc00 x1876699503962368/t0(0) o400->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760830.142278:0:2391:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c855e800 already not connected (conn 3, was 1): CONNECTING 00000100:00080000:2.0:1789760830.142319:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760830.142342:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000400:0.0:1789760830.142345:0:2391:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760814/real 1789760814] req@ffff9018d0f3e300 x1876699503962240/t0(0) o400->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760830.142361:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760830.142376:0:2390:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c7fd9800 already not connected (conn 3, was 1): CONNECTING 00000100:00080000:2.0:1789760830.142380:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760830.142386:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:1.0:1789760830.142393:0:2390:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760814/real 1789760814] req@ffff9018d0f3c700 x1876699503961984/t0(0) o400->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760830.142395:0:2391:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d0f3e300 x1876699503962240/t0(0) o400->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760830.142400:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760830.142429:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760830.142444:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760830.142450:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760830.142455:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760830.142456:0:2391:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c888a000 already not connected (conn 3, was 1): CONNECTING 00000100:00080000:2.0:1789760830.142463:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1789760830.142464:0:2390:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018d0f3c700 x1876699503961984/t0(0) o400->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760830.142477:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503961856:192.168.203.158@tcp:400:kworker.0 00000100:00080000:1.0:1789760830.142478:0:2390:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c855a000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:0.0:1789760830.142487:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:453b2994-2b6d-4a69-9aac-a9e10430c708:2391:1876699503962240:192.168.203.158@tcp:400:kworker.0 00000100:00100000:1.0:1789760830.142495:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:453b2994-2b6d-4a69-9aac-a9e10430c708:2390:1876699503962368:192.168.203.158@tcp:400:kworker.0 00000100:00100000:1.0:1789760830.142503:0:2390:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2390:1876699503961984:192.168.203.158@tcp:400:kworker.0 00010000:00010000:3.0:1789760830.782217:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760831.806284:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760832.830355:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760833.150325:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760833.150351:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760833.150357:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760833.150361:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760833.150365:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760833.150368:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760833.150372:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760833.150377:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760833.150382:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760833.150386:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760833.150390:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760833.150394:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1789760833.854438:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1789760834.238242:0:2391:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760818/real 1789760818] req@ffff9018c66a5180 x1876699503962752/t0(0) o400->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760834 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760834.253459:0:2391:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9018c66a5180 x1876699503962752/t0(0) o400->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760834 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760834.253471:0:2391:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9018c855a000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1789760834.253568:0:2391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2391:1876699503962752:192.168.203.158@tcp:400:kworker.0 00010000:00010000:3.0:1789760834.878363:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760835.902371:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760836.926319:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760837.950333:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760838.334210:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760838.334225:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760838.334230:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760838.334235:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760838.334239:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760838.334283:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760838.334298:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760838.334302:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760838.334306:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760838.334311:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760838.334315:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760838.334319:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760838.974304:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760839.998374:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760840.382173:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760825/real 1789760825] req@ffff9018c66a7800 x1876699503963392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760840.382247:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699503963392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760840 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760840.382281:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760840.382314:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760840.382320:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503963392:192.168.203.158@tcp:8:kworker.0 00000100:00080000:1.0:1789760840.382427:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760840.382437:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760840.382441:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760840.382446:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760840.382449:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760840.382453:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760840.382456:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760840.382460:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760840.382464:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760840.382475:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 756 00000100:00080000:1.0:1789760840.382493:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1789760840.382503:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760840.382520:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789760840.382568:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699503964160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760840.382607:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760840.382613:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760840.382617:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760840.382621:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760840.382683:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503964160:192.168.203.158@tcp:8:kworker.0 00010000:00010000:3.0:1789760841.022327:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760842.046343:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760843.070195:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760843.454174:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760828/real 1789760828] req@ffff9018c66a5500 x1876699503964032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760843.454216:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699503964032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760843 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760843.454240:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760828/real 1789760828] req@ffff9018c66a4e00 x1876699503963648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760843.454258:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699503963648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760843 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760843.454269:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760828/real 1789760828] req@ffff9018c66a6300 x1876699503963776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760843.454358:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699503963776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760843 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760843.454377:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760828/real 1789760828] req@ffff9018c66a4700 x1876699503963904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760843.454389:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699503963904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760843 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760843.454400:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760828/real 1789760828] req@ffff9018c66a6680 x1876699503963520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760843.454412:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699503963520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760843 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760843.454430:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760843.454449:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760843.454454:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503964032:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789760843.454460:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760843.454465:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760843.454470:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503963648:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789760843.454475:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760843.454481:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789760843.454486:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760843.454486:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503963776:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789760843.454495:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760843.454498:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760843.454498:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760843.454505:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760843.454508:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760843.454513:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 759 00000100:00080000:3.0:1789760843.454515:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760843.454520:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503963904:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789760843.454523:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1789760843.454530:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760843.454533:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789760843.454539:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:0.0:1789760843.454545:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760843.454545:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503963520:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789760843.454579:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699503964288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760843.454594:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760843.454597:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760843.454599:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760843.454602:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 759 00000100:00080000:0.0:1789760843.454604:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760843.454609:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760843.454610:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503964288:192.168.203.158@tcp:38:kworker.0 00000008:00100000:0.0:1789760843.454612:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760843.454624:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699503964416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760843.454631:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760843.454634:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760843.454636:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760843.454638:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760843.454639:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760843.454642:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 759 00000100:00080000:0.0:1789760843.454645:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760843.454647:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760843.454650:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760843.454657:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699503964544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760843.454663:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760843.454665:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760843.454666:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760843.454669:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 759 00000100:00080000:0.0:1789760843.454671:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760843.454675:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760843.454680:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760843.454693:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699503964672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760843.454696:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503964416:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789760843.454719:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760843.454723:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760843.454725:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760843.454730:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 759 00000100:00100000:3.0:1789760843.454733:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503964544:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789760843.454736:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760843.454743:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760843.454761:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503964672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789760843.454762:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699503964800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760843.454776:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760843.454779:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760843.454795:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503964800:192.168.203.158@tcp:250:kworker.0 00010000:00010000:3.0:1789760844.094214:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760845.118280:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760845.502157:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760845.502171:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760845.502184:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760845.502203:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760845.502206:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760845.502208:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760845.502210:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760845.502213:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760845.502215:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760845.502217:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760845.502219:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760845.502221:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1789760846.142308:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760847.166233:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760848.190231:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760848.510186:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760848.510196:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760848.510199:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760848.510203:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760848.510205:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760848.510208:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760848.510211:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760848.510214:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760848.510217:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760848.510220:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760848.510222:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760848.510226:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1789760849.214292:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760850.238319:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760851.262283:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760852.286395:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760853.310315:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760853.694285:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760853.694295:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760853.694299:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760853.694302:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760853.694305:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760853.694308:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760853.694310:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760853.694313:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760853.694316:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760853.694320:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760853.694323:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760853.694327:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760854.334289:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760855.358360:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760856.382320:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760857.406252:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760858.430291:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760858.814231:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760858.814241:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760858.814245:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760858.814248:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760858.814250:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760858.814253:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760858.814255:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760858.814258:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760858.814260:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760858.814263:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760858.814265:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760858.814267:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1789760859.454261:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760860.478274:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760860.862170:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1789760840/real 0] req@ffff9018c66a5180 x1876699503964160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760860 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760860.862204:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699503964160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760860 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1789760861.502280:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760862.526332:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760863.550299:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760863.934161:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1789760843/real 0] req@ffff9018c8b9ad80 x1876699503964800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760863 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760863.934185:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760863.934191:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699503964800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760863 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760863.934192:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760863.934198:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760863.934206:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760863.934212:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760863.934215:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1789760843/real 0] req@ffff9018c8b98380 x1876699503964672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760863 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760863.934216:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760863.934222:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760863.934230:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760863.934236:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760863.934240:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760863.934240:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699503964672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760863 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760863.934245:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760863.934252:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760863.934262:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1789760843/real 0] req@ffff9018c8b9a300 x1876699503964416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760863 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760863.934273:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699503964416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760863 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760863.934281:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1789760843/real 0] req@ffff9018c8b99500 x1876699503964544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760863 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760863.934289:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699503964544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760863 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760863.934295:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1789760843/real 0] req@ffff9018c8b99c00 x1876699503964288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760863 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760863.934302:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699503964288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760863 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1789760864.574251:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760865.598284:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760866.622287:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760867.646295:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760868.670249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760869.054217:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760869.054239:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760869.054243:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760869.054246:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760869.054249:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760869.054251:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760869.054254:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760869.054256:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760869.054258:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760869.054261:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760869.054263:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760869.054265:0:12619:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760869.694290:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760870.718222:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760871.742276:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760872.766176:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760873.790214:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760874.174160:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760874.174171:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760874.174175:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760874.174179:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760874.174181:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760874.174184:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760874.174185:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760874.174188:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760874.174191:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760874.174193:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760874.174207:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760874.174209:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1789760874.814140:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789760875.198213:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.158:988 00000400:00000100:2.0:1789760875.198255:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789760875.198269:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789760875.198299:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789760875.198308:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789760875.198315:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789760875.198323:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789760875.198328:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00080000:3.0:1789760875.198337:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760875.198364:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760875.198371:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503964672:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789760875.198379:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760875.198387:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760875.198392:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503964416:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789760875.198399:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760875.198405:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760875.198411:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760875.198414:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760875.198415:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789760875.198419:0:11323:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760875.198424:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760875.198424:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503964544:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789760875.198433:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789760875.198436:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760875.198437:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760875.198446:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 775 00000100:00080000:1.0:1789760875.198453:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1789760875.198456:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760875.198460:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503964288:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789760875.198463:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789760875.198468:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760875.198474:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000008:00100000:1.0:1789760875.198475:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760875.198478:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503964160:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789760875.198504:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699503964928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760875.198506:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760875.198521:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789760875.198522:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760875.198528:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789760875.198528:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503964800:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789760875.198531:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760875.198539:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 772 00000100:00080000:1.0:1789760875.198542:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1789760875.198545:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760875.198550:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789760875.198554:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503964928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789760875.198560:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699503965056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760875.198569:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760875.198572:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760875.198573:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760875.198575:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 775 00000100:00080000:1.0:1789760875.198577:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1789760875.198580:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760875.198583:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760875.198591:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699503965184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760875.198598:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760875.198600:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760875.198602:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789760875.198607:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789760875.198610:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 775 00000100:00080000:1.0:1789760875.198615:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1789760875.198618:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760875.198622:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760875.198630:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699503965312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760875.198640:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760875.198640:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760875/real 1789760875] req@ffff9018c66a4380 x1876699503964928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760875.198647:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760875.198650:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760875.198658:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 775 00000100:00080000:1.0:1789760875.198662:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1789760875.198665:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760875.198670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699503964928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760875.198676:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699503965440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760875.198689:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760875.198691:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760875.198696:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789760875.198697:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760875.198701:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760875.198703:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503964928:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789760875.198707:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 775 00000100:00080000:1.0:1789760875.198713:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1789760875.198716:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760875.198720:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760875.198723:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503965056:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789760875.198731:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699503965568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760875.198743:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760875.198755:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760875/real 1789760875] req@ffff9018c66a7100 x1876699503965056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760875.198764:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699503965056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760875.198770:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760875.198772:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760875.198774:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503965056:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760875.198779:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503965184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760875.198789:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760875.198796:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760875/real 1789760875] req@ffff9018c66a7480 x1876699503965184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760875.198803:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699503965184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760875.198808:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760875.198810:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760875.198812:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503965184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760875.198815:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503965312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760875.198824:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760875.198830:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760875/real 1789760875] req@ffff9018c66a4a80 x1876699503965312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760875.198837:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699503965312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760875.198842:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760875.198843:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760875.198845:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503965312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760875.198849:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503965440:192.168.203.158@tcp:250:kworker.0 00000800:00000100:3.0:1789760875.198858:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760875.198867:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760875/real 1789760875] req@ffff9018c66a6a00 x1876699503965440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760875.198875:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699503965440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760875.198880:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760875.198882:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760875.198884:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503965440:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760875.198894:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503965568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760875.198904:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760875.198910:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760875/real 1789760875] req@ffff9018c66a5880 x1876699503965568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760875.198917:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699503965568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760875.198922:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760875.198924:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760875.198926:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503965568:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789760875.838162:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760876.862172:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760877.886167:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760878.910254:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760879.934161:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760880.318150:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760880.318161:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760880.318166:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760880.318177:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 806 00000100:00080000:2.0:1789760880.318189:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760880.318197:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760880.318241:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699503965696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760880.318276:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760880.318280:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760880.318281:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760880.318285:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 806 00000100:00080000:2.0:1789760880.318287:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760880.318290:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760880.318301:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760880.318318:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699503965824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760880.318326:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760880.318333:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760880.318336:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760880.318336:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503965696:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789760880.318343:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 806 00000100:00080000:2.0:1789760880.318349:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760880.318354:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760880.318357:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760880.318368:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699503965952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760880.318375:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760880.318379:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760880.318380:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760880.318384:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 806 00000100:00080000:2.0:1789760880.318399:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760880.318403:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760880.318409:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789760880.318422:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699503966080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760880.318435:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760880.318439:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760880.318442:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760880.318447:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503965824:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789760880.318450:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 806 00000100:00080000:2.0:1789760880.318459:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760880.318465:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760880.318470:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760880.318478:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503965952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760880.318485:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699503966208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760880.318507:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760880.318511:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789760880.318514:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503966080:192.168.203.158@tcp:8:kworker.0 00000100:00080000:2.0:1789760880.318516:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760880.318527:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 806 00000100:00080000:2.0:1789760880.318534:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789760880.318539:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503966208:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789760880.318545:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760880.318552:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760880.318567:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699503966336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760880.318619:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503966336:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760880.958230:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760881.982257:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760883.006252:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789760883.390312:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.158:988 00000400:00000100:0.0:1789760883.390360:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789760883.390377:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760883.390419:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760883.390428:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760883.390435:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760883.390444:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760883.390452:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789760883.390483:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760880/real 1789760883] req@ffff9018d0f24380 x1876699503966336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760883.390514:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699503966336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760883.390529:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760883.390535:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760883.390541:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503966336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760883.390551:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760880/real 1789760883] req@ffff9018d0f25c00 x1876699503966208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760883.390565:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699503966208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760883.390574:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760883.390578:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760883.390582:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503966208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760883.390590:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760880/real 1789760883] req@ffff9018d0f25f80 x1876699503965824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760883.390610:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699503965824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760883.390619:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760883.390623:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760883.390626:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503965824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760883.390634:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760880/real 1789760883] req@ffff9018d0f24a80 x1876699503965952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760883.390646:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699503965952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760883.390654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760883.390657:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760883.390660:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503965952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760883.390667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760880/real 1789760883] req@ffff9018d0f25180 x1876699503966080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760883.390682:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699503966080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760883.390691:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760883.390695:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760883.390698:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503966080:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760883.390705:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760880/real 1789760883] req@ffff9018d0f27800 x1876699503965696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760883.390719:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699503965696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760883.390727:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760883.390731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760883.390734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503965696:192.168.203.158@tcp:250:kworker.0 00010000:00010000:3.0:1789760884.030250:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760885.054240:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760885.438114:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760885.438125:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760885.438129:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760885.438140:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 811 00000100:00080000:2.0:1789760885.438151:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760885.438159:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760885.438201:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699503966464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760885.438229:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760885.438233:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760885.438235:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760885.438240:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 811 00000100:00080000:2.0:1789760885.438244:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760885.438248:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760885.438261:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760885.438275:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699503966592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760885.438282:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503966464:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789760885.438286:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760885.438289:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760885.438291:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760885.438294:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 811 00000100:00080000:2.0:1789760885.438298:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760885.438301:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760885.438306:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760885.438315:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699503966720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760885.438323:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760885.438327:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760885.438328:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760885.438332:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 811 00000100:00080000:2.0:1789760885.438352:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760885.438356:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760885.438361:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789760885.438372:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699503966848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760885.438381:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760885.438385:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760885.438388:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760885.438392:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 811 00000100:00080000:2.0:1789760885.438395:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760885.438400:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760885.438406:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760885.438413:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503966592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760885.438416:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699503966976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760885.438425:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760885.438428:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760885.438430:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760885.438434:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 811 00000100:00080000:2.0:1789760885.438441:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760885.438448:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760885.438454:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503966720:192.168.203.158@tcp:38:kworker.0 00000008:00100000:2.0:1789760885.438456:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760885.438470:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699503967104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760885.438482:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503966848:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760885.438522:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503966976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760885.438569:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503967104:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760886.078203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789760886.462318:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.158:988 00000400:00000100:0.0:1789760886.462363:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789760886.462378:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760886.462415:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760886.462425:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760886.462435:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760886.462448:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789760886.462454:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760885/real 1789760886] req@ffff9018d0f26d80 x1876699503966592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1789760886.462461:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789760886.462485:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699503966592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760886.462494:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760886.462498:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760886.462502:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503966592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760886.462507:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760885/real 1789760886] req@ffff9018d0f24e00 x1876699503966720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760886.462515:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699503966720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760886.462519:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760886.462522:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760886.462524:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503966720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760886.462528:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760885/real 1789760886] req@ffff9018d0f27480 x1876699503966848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760886.462535:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699503966848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760886.462540:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760886.462542:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760886.462544:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503966848:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760886.462547:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760885/real 1789760886] req@ffff9018d0f24700 x1876699503966464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760886.462554:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699503966464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760886.462559:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760886.462561:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760886.462563:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503966464:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760886.462582:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760885/real 1789760886] req@ffff9018d0f26680 x1876699503966976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760886.462590:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699503966976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760886.462595:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760886.462597:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760886.462599:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503966976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760886.462602:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760885/real 1789760886] req@ffff9018d0f25500 x1876699503967104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760886.462609:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699503967104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760886.462613:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760886.462615:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760886.462617:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503967104:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760887.102180:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760888.126235:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760889.150262:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760890.174207:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760890.558162:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760890.558172:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760890.558177:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760890.558187:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 817 00000100:00080000:2.0:1789760890.558200:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760890.558207:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760890.558254:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699503967232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760890.558286:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760890.558289:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760890.558291:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760890.558294:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 817 00000100:00080000:2.0:1789760890.558296:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760890.558299:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760890.558308:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760890.558320:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699503967360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760890.558337:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760890.558342:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760890.558344:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760890.558350:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 817 00000100:00080000:2.0:1789760890.558354:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789760890.558364:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503967232:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789760890.558366:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760890.558385:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760890.558407:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699503967488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760890.558417:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760890.558420:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760890.558422:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760890.558426:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 817 00000100:00080000:2.0:1789760890.558439:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760890.558443:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760890.558448:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789760890.558457:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699503967616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760890.558464:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760890.558467:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760890.558469:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760890.558472:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 817 00000100:00080000:2.0:1789760890.558475:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760890.558482:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760890.558482:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503967360:192.168.203.158@tcp:38:kworker.0 00000008:00100000:2.0:1789760890.558494:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760890.558511:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699503967744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760890.558518:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760890.558521:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760890.558523:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760890.558525:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 817 00000100:00080000:2.0:1789760890.558530:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760890.558534:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760890.558536:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503967488:192.168.203.158@tcp:38:kworker.0 00000008:00100000:2.0:1789760890.558541:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760890.558559:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699503967872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760890.558570:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503967616:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760890.558605:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503967744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760890.558635:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503967872:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789760891.198248:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760892.222189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760893.246218:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789760893.630268:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.158:988 00000400:00000100:0.0:1789760893.630306:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789760893.630319:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760893.630351:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760893.630358:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760893.630365:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760893.630373:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789760893.630385:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789760893.630412:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760890/real 1789760893] req@ffff9018d0f24e00 x1876699503967872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760893.630441:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699503967872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760893.630453:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760893.630458:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760893.630463:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503967872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760893.630471:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760890/real 1789760893] req@ffff9018d0f26680 x1876699503967488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760893.630485:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699503967488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760893.630494:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760893.630498:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760893.630501:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503967488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760893.630509:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760890/real 1789760893] req@ffff9018d0f24700 x1876699503967616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760893.630529:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699503967616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760893.630538:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760893.630542:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760893.630545:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503967616:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760893.630553:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760890/real 1789760893] req@ffff9018d0f27480 x1876699503967744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760893.630566:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699503967744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760893.630573:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760893.630577:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760893.630581:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503967744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760893.630587:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760890/real 1789760893] req@ffff9018d0f24000 x1876699503967232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760893.630601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699503967232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760893.630609:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760893.630612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760893.630616:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503967232:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760893.630623:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760890/real 1789760893] req@ffff9018d0f25500 x1876699503967360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760893.630635:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699503967360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760893.630644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760893.630648:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760893.630651:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503967360:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789760894.270281:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760895.294221:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760895.678097:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760895.678109:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760895.678113:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760895.678123:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 822 00000100:00080000:2.0:1789760895.678135:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760895.678143:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760895.678182:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699503968000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760895.678208:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760895.678213:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760895.678215:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760895.678219:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 822 00000100:00080000:2.0:1789760895.678223:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760895.678228:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760895.678241:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760895.678254:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699503968128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760895.678264:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503968000:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789760895.678265:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760895.678269:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760895.678271:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760895.678274:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 822 00000100:00080000:2.0:1789760895.678278:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760895.678281:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760895.678287:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760895.678297:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699503968256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760895.678307:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760895.678310:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760895.678312:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760895.678315:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 822 00000100:00080000:2.0:1789760895.678337:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760895.678341:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760895.678346:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789760895.678349:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760895.678357:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699503968384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760895.678367:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760895.678370:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760895.678372:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760895.678375:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 822 00000100:00080000:2.0:1789760895.678378:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760895.678381:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760895.678385:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760895.678395:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699503968512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760895.678399:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760895/real 1789760895] req@ffff9018d0f26d80 x1876699503968000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760895.678404:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760895.678408:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760895.678410:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760895.678414:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 822 00000100:00080000:2.0:1789760895.678417:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789760895.678420:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699503968000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760895.678421:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760895.678426:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789760895.678432:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789760895.678435:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699503968640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760895.678436:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760895.678441:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503968000:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760895.678471:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503968128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760895.678498:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760895.678511:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760895/real 1789760895] req@ffff9018d0f27800 x1876699503968128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760895.678524:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699503968128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760895.678547:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760895.678550:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760895.678554:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503968128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760895.678561:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503968256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760895.678584:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760895.678596:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760895/real 1789760895] req@ffff9018d0f25180 x1876699503968256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760895.678620:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699503968256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760895.678628:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760895.678631:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760895.678634:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503968256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760895.678640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503968384:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760895.678667:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760895.678679:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760895/real 1789760895] req@ffff9018d0f24a80 x1876699503968384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760895.678697:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699503968384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760895.678706:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760895.678709:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760895.678714:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503968384:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760895.678721:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503968512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760895.678743:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760895.678756:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760895/real 1789760895] req@ffff9018d0f25f80 x1876699503968512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760895.678769:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699503968512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760895.678778:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760895.678781:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760895.678785:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503968512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760895.678792:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503968640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760895.678835:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760895.678849:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760895/real 1789760895] req@ffff9018d0f25c00 x1876699503968640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760895.678862:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699503968640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760895.678869:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760895.678872:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760895.678875:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503968640:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760896.318161:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760897.342163:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760898.366208:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760899.390163:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760900.414287:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760900.798206:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760900.798221:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760900.798227:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760900.798243:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 827 00000100:00080000:0.0:1789760900.798260:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760900.798271:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789760900.798330:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699503968768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760900.798371:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760900.798376:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760900.798380:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760900.798386:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 827 00000100:00080000:0.0:1789760900.798391:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760900.798396:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760900.798410:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760900.798419:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503968768:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789760900.798442:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699503968896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760900.798459:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760900.798464:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760900.798467:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760900.798474:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 827 00000100:00080000:0.0:1789760900.798478:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760900.798483:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760900.798489:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760900.798505:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699503969024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760900.798517:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789760900.798538:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760900.798545:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760900.798548:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760900.798554:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 827 00000100:00080000:0.0:1789760900.798570:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760900.798575:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760900.798582:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789760900.798592:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760900/real 1789760900] req@ffff9018c8b9b100 x1876699503968768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760900.798601:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699503969152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760900.798663:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760900.798668:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1789760900.798674:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699503968768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760900.798675:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760900.798694:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 827 00000100:00080000:0.0:1789760900.798705:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760900.798711:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760900.798715:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760900.798737:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789760900.798740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760900.798752:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503968768:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789760900.798760:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699503969280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760900.798797:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760900.798803:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503968896:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789760900.798804:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760900.798811:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760900.798832:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 827 00000100:00080000:0.0:1789760900.798843:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760900.798848:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760900.798855:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760900.798871:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699503969408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760900.798881:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760900.798905:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760900/real 1789760900] req@ffff9018c8b9aa00 x1876699503968896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760900.798916:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699503968896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760900.798923:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760900.798930:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760900.798933:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503968896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760900.798939:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503969024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760900.798954:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760900.798961:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760900/real 1789760900] req@ffff9018c8b9b800 x1876699503969024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760900.798969:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699503969024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760900.798974:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760900.798976:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760900.798978:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503969024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760900.798982:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503969152:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760900.798993:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760900.798999:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760900/real 1789760900] req@ffff9018c8b9b480 x1876699503969152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760900.799006:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699503969152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760900.799011:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760900.799013:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760900.799015:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503969152:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760900.799026:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503969280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760900.799037:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760900.799043:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760900/real 1789760900] req@ffff9018c8b9a680 x1876699503969280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760900.799069:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699503969280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760900.799080:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760900.799082:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760900.799084:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503969280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760900.799089:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503969408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760900.799101:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760900.799109:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760900/real 1789760900] req@ffff9018c8b98000 x1876699503969408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760900.799117:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699503969408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760900.799122:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760900.799124:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760900.799126:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503969408:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760901.438222:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760902.462266:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760903.486222:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760904.510377:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760905.534144:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760905.918129:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760905.918138:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760905.918143:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760905.918154:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 832 00000100:00080000:0.0:1789760905.918165:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760905.918173:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789760905.918210:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699503969536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760905.918246:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760905.918252:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760905.918254:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760905.918259:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 832 00000100:00080000:0.0:1789760905.918264:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760905.918270:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760905.918280:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760905.918297:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699503969664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760905.918298:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503969536:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789760905.918309:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760905.918314:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760905.918315:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760905.918319:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 832 00000100:00080000:0.0:1789760905.918322:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760905.918326:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760905.918330:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760905.918340:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699503969792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760905.918348:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760905.918351:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760905.918352:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760905.918355:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 832 00000100:00080000:0.0:1789760905.918357:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760905.918360:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760905.918365:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789760905.918373:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699503969920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760905.918381:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760905.918383:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760905.918384:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760905.918387:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 832 00000100:00080000:0.0:1789760905.918388:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760905.918391:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760905.918394:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760905.918400:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699503970048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760905.918405:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760905.918408:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760905.918409:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760905.918412:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 832 00000100:00080000:0.0:1789760905.918415:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760905.918418:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760905.918421:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760905.918432:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699503970176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760905.918449:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503969664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760905.918476:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503969792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760905.918490:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503969920:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789760905.918503:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503970048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760905.918517:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503970176:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789760906.558164:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760907.582305:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760908.606159:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789760908.990459:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.158:988 00000400:00000100:3.0:1789760908.990499:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789760908.990516:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789760908.990583:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789760908.990590:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789760908.990596:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789760908.990601:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789760908.990606:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:2.0:1789760908.990676:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760905/real 1789760908] req@ffff9018c8b98e00 x1876699503969664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760908.990704:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699503969664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760908.990831:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760908.990838:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760908.990845:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503969664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760908.990858:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760905/real 1789760908] req@ffff9018c8b99880 x1876699503969792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760908.990873:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699503969792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760908.990882:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760908.990886:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760908.990890:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503969792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760908.990899:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760905/real 1789760908] req@ffff9018c8b99f80 x1876699503969920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760908.990911:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699503969920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760908.990919:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760908.990924:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760908.990928:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503969920:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789760908.990936:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760905/real 1789760908] req@ffff9018c8b9bb80 x1876699503970048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760908.990949:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699503970048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760908.990958:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760908.990962:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760908.990973:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503970048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760908.990982:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760905/real 1789760908] req@ffff9018c8b98700 x1876699503970176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760908.990996:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699503970176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760908.991005:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760908.991008:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760908.991012:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503970176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760908.991076:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760905/real 1789760908] req@ffff9018c8b98a80 x1876699503969536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760908.991094:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699503969536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760908.991104:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760908.991107:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760908.991111:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503969536:192.168.203.158@tcp:250:kworker.0 00010000:00010000:0.0:1789760909.630188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760910.654191:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760911.038174:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760911.038196:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760911.038201:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760911.038215:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 837 00000100:00080000:2.0:1789760911.038226:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760911.038236:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760911.038282:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699503970304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760911.038325:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760911.038330:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760911.038332:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760911.038340:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 837 00000100:00080000:2.0:1789760911.038345:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760911.038350:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760911.038372:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760911.038389:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699503970432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760911.038389:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503970304:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789760911.038423:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760911.038437:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760911.038438:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760911.038441:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 837 00000100:00080000:2.0:1789760911.038444:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760911.038447:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760911.038450:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760911.038460:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699503970560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760911.038466:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760911.038469:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760911.038470:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760911.038474:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 837 00000100:00080000:2.0:1789760911.038477:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760911.038479:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760911.038483:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789760911.038502:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699503970688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760911.038508:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760911.038511:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760911.038512:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760911.038515:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 837 00000100:00080000:2.0:1789760911.038525:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760911.038528:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760911.038531:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789760911.038538:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760911.038546:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699503970816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760911.038568:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760911.038572:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760911.038574:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760911.038580:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 837 00000100:00080000:2.0:1789760911.038585:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760911.038590:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760911.038597:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760911.038616:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760911/real 1789760911] req@ffff9018d0f24380 x1876699503970304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760911.038617:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699503970944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760911.038687:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699503970304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760911.038724:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760911.038731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760911.038736:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503970304:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760911.038771:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503970432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760911.038802:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760911.038817:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760911/real 1789760911] req@ffff9018d0f27100 x1876699503970432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760911.038833:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699503970432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760911.038841:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760911.038845:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760911.038848:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503970432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760911.038855:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503970560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760911.038879:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760911.038891:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760911/real 1789760911] req@ffff9018d0f26300 x1876699503970560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760911.038903:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699503970560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760911.038910:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760911.038914:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760911.038917:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503970560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760911.038925:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503970688:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760911.038948:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760911.038961:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760911/real 1789760911] req@ffff9018d0f26a00 x1876699503970688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760911.038972:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699503970688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760911.038982:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760911.038986:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760911.038989:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503970688:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760911.038996:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503970816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760911.039019:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760911.039031:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760911/real 1789760911] req@ffff9018d0f25c00 x1876699503970816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760911.039085:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699503970816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760911.039093:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760911.039097:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760911.039100:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503970816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760911.039107:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503970944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760911.039130:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760911.039142:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760911/real 1789760911] req@ffff9018d0f25f80 x1876699503970944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760911.039245:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699503970944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760911.039255:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760911.039260:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760911.039264:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503970944:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760911.678234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760912.702196:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760913.726270:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760914.750355:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760915.774253:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760916.158206:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760916.158215:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760916.158218:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760916.158227:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 842 00000100:00080000:1.0:1789760916.158236:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760916.158242:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789760916.158277:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699503971072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760916.158301:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760916.158304:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760916.158306:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760916.158309:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 842 00000100:00080000:1.0:1789760916.158311:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760916.158314:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760916.158322:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760916.158332:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699503971200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760916.158340:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760916.158342:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760916.158344:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760916.158347:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 842 00000100:00080000:1.0:1789760916.158349:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760916.158351:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760916.158355:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760916.158366:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503971072:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789760916.158368:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699503971328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760916.158393:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760916.158396:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760916.158397:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760916.158400:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 842 00000100:00080000:1.0:1789760916.158403:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760916.158406:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760916.158409:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789760916.158421:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699503971456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760916.158428:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760916.158431:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760916.158432:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760916.158434:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 842 00000100:00080000:1.0:1789760916.158436:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760916.158440:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760916.158442:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760916.158450:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699503971584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760916.158456:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760916.158458:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760916.158460:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760916.158463:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 842 00000100:00080000:1.0:1789760916.158466:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760916.158469:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760916.158472:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789760916.158477:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789760916.158481:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699503971712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760916.158556:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760916/real 1789760916] req@ffff9018c66a5880 x1876699503971072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760916.158581:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699503971072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760916.158591:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760916.158594:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760916.158598:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503971072:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760916.158605:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503971200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760916.158624:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760916.158634:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760916/real 1789760916] req@ffff9018c66a6a00 x1876699503971200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760916.158643:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699503971200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760916.158648:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760916.158651:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760916.158653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503971200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760916.158700:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503971328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760916.158732:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760916.158739:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760916/real 1789760916] req@ffff9018c66a4a80 x1876699503971328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760916.158748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699503971328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760916.158758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760916.158760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760916.158763:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503971328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760916.158767:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503971456:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760916.158779:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760916.158801:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760916/real 1789760916] req@ffff9018c66a7480 x1876699503971456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760916.158809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699503971456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760916.158815:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760916.158817:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760916.158819:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503971456:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760916.158823:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503971584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760916.158836:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760916.158842:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760916/real 1789760916] req@ffff9018c66a7100 x1876699503971584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760916.158850:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699503971584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760916.158855:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760916.158857:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760916.158860:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503971584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760916.158864:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503971712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760916.158875:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760916.158881:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760916/real 1789760916] req@ffff9018c66a4380 x1876699503971712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760916.158888:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699503971712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760916.158894:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760916.158896:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760916.158898:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503971712:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760916.798215:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760917.822157:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760918.846200:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760919.870276:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760920.894165:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760921.278218:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760921.278240:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760921.278245:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760921.278271:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 847 00000100:00080000:0.0:1789760921.278283:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760921.278291:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789760921.278354:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699503971840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760921.278382:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760921.278386:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760921.278388:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760921.278392:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 847 00000100:00080000:0.0:1789760921.278396:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760921.278401:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760921.278429:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760921.278443:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699503971968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760921.278456:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760921.278463:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789760921.278464:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503971840:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789760921.278465:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760921.278495:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 847 00000100:00080000:0.0:1789760921.278498:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760921.278500:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760921.278503:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760921.278513:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699503972096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760921.278519:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760921.278522:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760921.278523:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760921.278526:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 847 00000100:00080000:0.0:1789760921.278530:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760921.278533:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760921.278540:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789760921.278547:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789760921.278554:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699503972224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760921.278574:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760921.278578:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760921.278582:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760921.278583:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760921/real 1789760921] req@ffff9018c8b98a80 x1876699503971840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760921.278590:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 847 00000100:00080000:0.0:1789760921.278596:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760921.278601:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760921.278606:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760921.278608:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699503971840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760921.278617:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699503972352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760921.278631:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760921.278639:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789760921.278646:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760921.278646:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503971840:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789760921.278654:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760921.278656:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760921.278660:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 847 00000100:00080000:0.0:1789760921.278663:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760921.278666:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760921.278669:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760921.278676:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503971968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789760921.278680:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699503972480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760921.278718:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760921.278733:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760921/real 1789760921] req@ffff9018c8b98700 x1876699503971968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760921.278743:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699503971968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760921.278749:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760921.278751:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760921.278754:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503971968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760921.278758:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503972096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760921.278780:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760921.278786:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760921/real 1789760921] req@ffff9018c8b9bb80 x1876699503972096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760921.278794:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699503972096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760921.278799:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760921.278801:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760921.278803:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503972096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760921.278807:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503972224:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760921.278818:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760921.278824:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760921/real 1789760921] req@ffff9018c8b99f80 x1876699503972224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760921.278832:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699503972224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760921.278837:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760921.278839:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760921.278841:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503972224:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760921.278845:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503972352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760921.278855:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760921.278860:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760921/real 1789760921] req@ffff9018c8b99880 x1876699503972352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760921.278868:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699503972352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760921.278873:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760921.278875:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760921.278878:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503972352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760921.278895:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503972480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760921.278915:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760921.278927:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760921/real 1789760921] req@ffff9018c8b98e00 x1876699503972480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760921.278937:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699503972480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760921.278947:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760921.278949:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760921.278951:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503972480:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789760921.918269:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760922.942283:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760923.966229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760924.990345:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760926.014187:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760926.398091:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760926.398100:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760926.398104:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760926.398111:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 852 00000100:00080000:0.0:1789760926.398120:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760926.398126:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789760926.398157:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699503972608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760926.398178:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760926.398181:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760926.398182:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760926.398185:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 852 00000100:00080000:0.0:1789760926.398187:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760926.398190:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760926.398198:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760926.398205:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699503972736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760926.398213:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760926.398216:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760926.398217:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760926.398219:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 852 00000100:00080000:0.0:1789760926.398221:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760926.398223:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760926.398226:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760926.398235:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503972608:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789760926.398237:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699503972864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760926.398259:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760926.398262:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760926.398264:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760926.398269:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 852 00000100:00080000:0.0:1789760926.398274:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760926.398278:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760926.398284:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789760926.398297:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699503972992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760926.398304:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760926.398307:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760926.398308:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760926.398311:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 852 00000100:00080000:0.0:1789760926.398315:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760926.398317:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760926.398321:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760926.398331:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699503973120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760926.398331:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503972736:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789760926.398344:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760926.398347:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760926.398348:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760926.398352:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 852 00000100:00080000:0.0:1789760926.398355:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760926.398358:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760926.398361:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760926.398369:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503972864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789760926.398373:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699503973248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760926.398391:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503972992:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760926.398407:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503973120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760926.398430:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503973248:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760927.038236:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760928.062314:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760929.086307:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789760929.470299:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:1.0:1789760929.470337:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789760929.470351:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789760929.470389:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789760929.470402:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789760929.470416:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789760929.470425:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760926/real 1789760929] req@ffff9018c8b9b480 x1876699503972864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1789760929.470431:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789760929.470443:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789760929.470461:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699503972864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760929.470476:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760929.470482:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760929.470487:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503972864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760929.470495:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760926/real 1789760929] req@ffff9018c8b9b800 x1876699503972992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760929.470510:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699503972992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760929.470519:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760929.470522:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760929.470527:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503972992:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760929.470534:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760926/real 1789760929] req@ffff9018c8b9aa00 x1876699503973120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760929.470547:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699503973120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760929.470553:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760929.470555:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760929.470570:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503973120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760929.470578:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760926/real 1789760929] req@ffff9018c8b98000 x1876699503972608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760929.470593:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699503972608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760929.470601:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760929.470604:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760929.470608:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503972608:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760929.470615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760926/real 1789760929] req@ffff9018c8b9a680 x1876699503972736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760929.470626:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699503972736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760929.470634:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760929.470643:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760929.470647:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503972736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760929.470677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760926/real 1789760929] req@ffff9018c8b9b100 x1876699503973248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760929.470689:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699503973248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760929.470696:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760929.470700:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760929.470703:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503973248:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760930.110268:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760931.134188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760931.518136:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760931.518145:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760931.518148:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760931.518157:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 858 00000100:00080000:0.0:1789760931.518166:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760931.518173:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789760931.518214:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699503973376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760931.518236:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760931.518239:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760931.518240:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760931.518243:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 858 00000100:00080000:0.0:1789760931.518246:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760931.518248:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760931.518256:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760931.518264:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699503973504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760931.518281:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760931.518284:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760931.518285:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760931.518289:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 858 00000100:00080000:0.0:1789760931.518291:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760931.518294:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760931.518297:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760931.518305:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699503973632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760931.518312:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760931.518316:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789760931.518316:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503973376:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789760931.518319:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760931.518326:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 858 00000100:00080000:0.0:1789760931.518329:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760931.518332:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760931.518336:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789760931.518350:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699503973760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760931.518360:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760931.518365:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760931.518367:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760931.518371:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 858 00000100:00080000:0.0:1789760931.518376:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760931.518381:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760931.518386:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789760931.518388:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789760931.518401:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699503973888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760931.518411:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760931.518415:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760931.518417:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760931.518421:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 858 00000100:00080000:0.0:1789760931.518425:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760931.518430:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760931.518435:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760931.518446:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760931/real 1789760931] req@ffff9018c8b9ad80 x1876699503973376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760931.518451:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699503974016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760931.518484:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699503973376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760931.518503:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760931.518508:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760931.518512:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503973376:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760931.518519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503973504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760931.518561:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760931.518578:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760931/real 1789760931] req@ffff9018c8b99c00 x1876699503973504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760931.518590:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699503973504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760931.518598:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760931.518601:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760931.518605:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503973504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760931.518633:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503973632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760931.518655:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760931.518667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760931/real 1789760931] req@ffff9018c8b99500 x1876699503973632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760931.518679:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699503973632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760931.518687:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760931.518691:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760931.518694:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503973632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760931.518701:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503973760:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760931.518722:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760931.518733:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760931/real 1789760931] req@ffff9018c8b9a300 x1876699503973760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760931.518746:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699503973760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760931.518755:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760931.518759:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760931.518762:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503973760:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760931.518768:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503973888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760931.518789:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760931.518800:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760931/real 1789760931] req@ffff9018c8b98380 x1876699503973888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760931.518813:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699503973888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760931.518821:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760931.518824:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760931.518828:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503973888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760931.518834:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503974016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760931.518855:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760931.518868:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760931/real 1789760931] req@ffff9018c8b9b100 x1876699503974016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760931.518880:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699503974016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760931.518888:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760931.518891:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760931.518894:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503974016:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760932.158215:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760933.182186:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760934.206230:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760935.230217:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760936.254176:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760936.638215:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760936.638226:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760936.638231:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760936.638251:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 863 00000100:00080000:2.0:1789760936.638267:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760936.638277:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760936.638329:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699503974144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760936.638361:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760936.638366:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760936.638384:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760936.638390:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 863 00000100:00080000:2.0:1789760936.638395:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760936.638402:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760936.638416:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760936.638435:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699503974272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760936.638449:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503974144:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789760936.638453:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760936.638465:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760936.638468:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760936.638488:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 863 00000100:00080000:2.0:1789760936.638494:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760936.638500:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760936.638537:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760936.638552:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699503974400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760936.638564:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789760936.638565:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760936.638571:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760936.638574:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760936.638581:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 863 00000100:00080000:2.0:1789760936.638594:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760936.638600:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760936.638607:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789760936.638612:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760936/real 1789760936] req@ffff9018d0f24a80 x1876699503974144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760936.638624:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699503974528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760936.638634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699503974144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760936.638640:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760936.638647:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760936.638648:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760936.638650:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760936.638653:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789760936.638661:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 863 00000100:00100000:3.0:1789760936.638661:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503974144:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789760936.638669:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760936.638676:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760936.638682:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760936.638692:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503974272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760936.638698:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699503974656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760936.638713:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789760936.638718:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789760936.638721:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760936.638726:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760936.638732:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760936/real 1789760936] req@ffff9018d0f25180 x1876699503974272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760936.638733:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 863 00000100:00080000:2.0:1789760936.638741:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760936.638751:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760936.638759:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699503974272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789760936.638760:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789760936.638772:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760936.638774:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760936.638778:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503974272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760936.638783:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699503974784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760936.638787:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503974400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760936.638821:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760936.638831:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760936/real 1789760936] req@ffff9018d0f27800 x1876699503974400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760936.638840:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699503974400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760936.638846:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760936.638849:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760936.638851:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503974400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760936.638855:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503974528:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760936.638866:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760936.638872:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760936/real 1789760936] req@ffff9018d0f26d80 x1876699503974528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760936.638880:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699503974528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760936.638890:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760936.638892:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760936.638895:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503974528:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760936.638905:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503974656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760936.638916:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760936.638922:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760936/real 1789760936] req@ffff9018d0f25500 x1876699503974656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760936.638932:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699503974656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760936.638941:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760936.638945:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760936.638948:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503974656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760936.638955:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503974784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760936.638976:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760936.638990:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760936/real 1789760936] req@ffff9018d0f24000 x1876699503974784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760936.639004:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699503974784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760936.639014:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760936.639017:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760936.639022:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503974784:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789760937.278275:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760938.302231:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760939.326267:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760940.350179:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760941.374220:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760941.758105:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760941.758119:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760941.758124:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760941.758138:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 868 00000100:00080000:2.0:1789760941.758151:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760941.758160:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760941.758206:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699503974912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760941.758241:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760941.758246:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760941.758248:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760941.758253:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 868 00000100:00080000:2.0:1789760941.758257:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760941.758263:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760941.758275:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760941.758288:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503974912:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789760941.758292:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699503975040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760941.758314:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760941.758317:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760941.758318:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760941.758323:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 868 00000100:00080000:2.0:1789760941.758327:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760941.758332:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760941.758339:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760941.758352:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699503975168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760941.758363:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760941.758367:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760941.758369:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760941.758375:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 868 00000100:00080000:2.0:1789760941.758379:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789760941.758385:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789760941.758385:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760941.758393:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789760941.758406:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699503975296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760941.758417:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760941.758420:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760941.758422:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760941.758426:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 868 00000100:00080000:2.0:1789760941.758430:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760941.758435:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760941.758441:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760941.758453:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760941/real 1789760941] req@ffff9018d0f27480 x1876699503974912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760941.758456:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699503975424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760941.758491:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760941.758496:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760941.758499:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760941.758506:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 868 00000100:00100000:0.0:1789760941.758507:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699503974912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760941.758513:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760941.758520:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760941.758528:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789760941.758542:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789760941.758546:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699503975552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760941.758552:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789760941.758560:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503974912:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789760941.758586:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503975040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789760941.758609:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789760941.758618:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760941/real 1789760941] req@ffff9018d0f24700 x1876699503975040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760941.758627:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699503975040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760941.758633:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760941.758636:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789760941.758638:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503975040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789760941.758642:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503975168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789760941.758660:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789760941.758672:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760941/real 1789760941] req@ffff9018d0f26680 x1876699503975168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760941.758684:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699503975168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760941.758693:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760941.758697:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789760941.758701:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503975168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789760941.758707:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503975296:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789760941.758729:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789760941.758741:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760941/real 1789760941] req@ffff9018d0f24e00 x1876699503975296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760941.758755:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699503975296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760941.758764:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760941.758767:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789760941.758771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503975296:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789760941.758777:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503975424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789760941.758797:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789760941.758807:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760941/real 1789760941] req@ffff9018d0f24000 x1876699503975424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760941.758815:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699503975424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760941.758822:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760941.758824:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789760941.758827:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503975424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789760941.758831:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503975552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789760941.758845:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789760941.758853:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760941/real 1789760941] req@ffff9018d0f25500 x1876699503975552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760941.758864:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699503975552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760941.758870:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760941.758872:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789760941.758874:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503975552:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760942.398204:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760943.422216:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760944.446199:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760945.470278:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760946.494246:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760946.878176:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760946.878185:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760946.878189:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760946.878199:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 873 00000100:00080000:0.0:1789760946.878209:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760946.878216:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789760946.878249:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699503975680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760946.878280:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760946.878284:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760946.878285:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760946.878289:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 873 00000100:00080000:0.0:1789760946.878292:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760946.878295:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760946.878305:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760946.878320:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699503975808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760946.878338:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760946.878340:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503975680:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789760946.878343:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760946.878348:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760946.878353:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 873 00000100:00080000:0.0:1789760946.878359:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760946.878362:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760946.878370:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760946.878381:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699503975936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760946.878390:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760946.878393:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760946.878395:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760946.878398:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 873 00000100:00080000:0.0:1789760946.878401:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760946.878406:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760946.878410:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789760946.878418:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789760946.878423:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699503976064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760946.878445:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760946.878450:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760946.878452:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760946.878457:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 873 00000100:00080000:0.0:1789760946.878461:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760946.878466:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760946.878473:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760946.878483:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760946/real 1789760946] req@ffff9018c8b9a680 x1876699503975680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760946.878488:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699503976192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760946.878513:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760946.878517:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760946.878519:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760946.878525:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699503975680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760946.878527:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 873 00000100:00080000:0.0:1789760946.878535:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760946.878543:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789760946.878547:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789760946.878550:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789760946.878555:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760946.878561:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503975680:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789760946.878567:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699503976320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760946.878587:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503975808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760946.878612:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760946.878624:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760946/real 1789760946] req@ffff9018c8b98000 x1876699503975808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760946.878636:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699503975808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760946.878644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760946.878648:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760946.878651:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503975808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760946.878658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503975936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760946.878674:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760946.878684:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760946/real 1789760946] req@ffff9018c8b9aa00 x1876699503975936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760946.878694:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699503975936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760946.878701:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760946.878704:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760946.878707:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503975936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760946.878712:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503976064:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760946.878729:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760946.878738:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760946/real 1789760946] req@ffff9018c8b9b800 x1876699503976064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760946.878748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699503976064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760946.878756:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760946.878758:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760946.878761:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503976064:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760946.878766:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503976192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760946.878783:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760946.878794:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760946/real 1789760946] req@ffff9018c8b9b480 x1876699503976192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760946.878805:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699503976192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760946.878812:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760946.878815:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760946.878818:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503976192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760946.878823:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503976320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760946.878837:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760946.878845:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760946/real 1789760946] req@ffff9018c8b98e00 x1876699503976320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760946.878856:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699503976320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760946.878872:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760946.878875:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760946.878879:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503976320:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760947.518205:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760948.542255:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760949.566199:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760950.590252:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760951.614182:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760951.998146:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760951.998155:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760951.998160:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760951.998170:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 878 00000100:00080000:2.0:1789760951.998180:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760951.998187:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760951.998220:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699503976448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760951.998246:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760951.998253:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760951.998255:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760951.998259:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 878 00000100:00080000:2.0:1789760951.998262:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760951.998266:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760951.998275:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760951.998288:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699503976576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760951.998299:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760951.998303:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760951.998306:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760951.998310:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 878 00000100:00080000:2.0:1789760951.998314:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789760951.998316:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503976448:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789760951.998321:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760951.998331:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760951.998346:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699503976704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760951.998357:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760951.998361:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760951.998363:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760951.998367:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 878 00000100:00080000:2.0:1789760951.998372:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760951.998376:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789760951.998382:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789760951.998384:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789760951.998412:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699503976832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760951.998424:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760951.998428:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760951.998437:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760951.998438:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760951/real 1789760951] req@ffff9018d0f26d80 x1876699503976448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760951.998446:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 878 00000100:00080000:2.0:1789760951.998454:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760951.998460:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760951.998464:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699503976448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789760951.998467:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789760951.998480:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760951.998485:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760951.998486:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699503976960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760951.998489:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503976448:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760951.998502:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503976576:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789760951.998507:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760951.998512:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760951.998514:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760951.998520:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 878 00000100:00080000:2.0:1789760951.998524:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789760951.998527:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789760951.998532:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760951.998540:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760951.998540:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760951/real 1789760951] req@ffff9018d0f27800 x1876699503976576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760951.998559:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699503976576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760951.998561:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699503977088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760951.998578:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760951.998585:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760951.998590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503976576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760951.998611:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503976704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760951.998637:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760951.998648:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760951/real 1789760951] req@ffff9018d0f25180 x1876699503976704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760951.998668:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699503976704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760951.998677:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760951.998681:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760951.998684:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503976704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760951.998690:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503976832:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760951.998707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760951.998715:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760951/real 1789760951] req@ffff9018d0f24a80 x1876699503976832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760951.998723:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699503976832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760951.998728:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760951.998730:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760951.998732:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503976832:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760951.998736:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503976960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760951.998745:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760951.998751:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760951/real 1789760951] req@ffff9018d0f25f80 x1876699503976960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760951.998758:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699503976960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760951.998767:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760951.998769:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760951.998771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503976960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760951.998774:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503977088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760951.998783:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760951.998788:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760951/real 1789760951] req@ffff9018d0f25c00 x1876699503977088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760951.998795:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699503977088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760951.998800:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760951.998802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760951.998803:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503977088:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789760952.638220:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760953.662162:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760954.686240:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760955.710146:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760956.734257:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760957.118110:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760957.118118:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760957.118121:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760957.118130:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 883 00000100:00080000:2.0:1789760957.118139:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760957.118144:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760957.118177:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699503977216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760957.118198:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760957.118201:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760957.118202:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760957.118206:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 883 00000100:00080000:2.0:1789760957.118208:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760957.118210:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760957.118218:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760957.118226:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699503977344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760957.118234:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760957.118239:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789760957.118240:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503977216:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789760957.118241:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760957.118249:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 883 00000100:00080000:2.0:1789760957.118251:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760957.118254:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760957.118257:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760957.118265:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699503977472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760957.118275:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760957.118278:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760957.118280:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760957.118282:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 883 00000100:00080000:2.0:1789760957.118292:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760957.118295:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760957.118299:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789760957.118301:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760957.118311:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699503977600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760957.118319:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760957.118321:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760957.118323:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760957.118325:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 883 00000100:00080000:2.0:1789760957.118327:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760957.118330:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760957.118334:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760957.118339:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760957/real 1789760957] req@ffff9018d0f26a00 x1876699503977216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760957.118343:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699503977728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760957.118366:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760957.118373:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760957.118377:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760957.118380:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 883 00000100:00080000:2.0:1789760957.118384:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760957.118387:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760957.118390:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760957.118392:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699503977216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760957.118401:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699503977856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760957.118421:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760957.118427:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760957.118431:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503977216:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760957.118453:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503977344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760957.118471:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760957.118480:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760957/real 1789760957] req@ffff9018d0f26300 x1876699503977344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760957.118489:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699503977344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760957.118495:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760957.118497:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760957.118500:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503977344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760957.118504:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503977472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760957.118514:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760957.118520:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760957/real 1789760957] req@ffff9018d0f27100 x1876699503977472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760957.118527:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699503977472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760957.118532:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760957.118534:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760957.118536:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503977472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760957.118540:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503977600:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760957.118550:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760957.118556:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760957/real 1789760957] req@ffff9018d0f24380 x1876699503977600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760957.118563:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699503977600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760957.118569:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760957.118571:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760957.118572:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503977600:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760957.118576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503977728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760957.118585:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760957.118591:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760957/real 1789760957] req@ffff9018d0f25c00 x1876699503977728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760957.118597:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699503977728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760957.118602:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760957.118604:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760957.118606:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503977728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760957.118609:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503977856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760957.118618:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760957.118624:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760957/real 1789760957] req@ffff9018d0f25f80 x1876699503977856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760957.118632:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699503977856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760957.118637:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760957.118638:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760957.118640:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503977856:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760957.758185:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760958.782251:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760959.806180:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760960.830239:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760961.854294:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760962.238190:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760962.238198:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760962.238201:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760962.238209:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 888 00000100:00080000:0.0:1789760962.238218:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760962.238224:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789760962.238255:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699503977984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760962.238277:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760962.238280:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760962.238282:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760962.238284:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 888 00000100:00080000:0.0:1789760962.238287:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760962.238289:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760962.238296:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760962.238303:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699503978112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760962.238310:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760962.238312:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760962.238327:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760962.238330:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 888 00000100:00080000:0.0:1789760962.238333:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760962.238336:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760962.238339:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760962.238348:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699503978240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760962.238356:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760962.238360:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760962.238363:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760962.238363:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503977984:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789760962.238372:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 888 00000100:00080000:0.0:1789760962.238374:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760962.238376:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760962.238380:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789760962.238389:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699503978368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760962.238396:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760962.238398:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760962.238399:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760962.238402:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 888 00000100:00080000:0.0:1789760962.238405:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760962.238407:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760962.238410:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760962.238417:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699503978496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760962.238423:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760962.238426:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760962.238427:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760962.238429:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 888 00000100:00080000:0.0:1789760962.238431:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760962.238433:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760962.238435:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760962.238444:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699503978624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760962.238459:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503978112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760962.238493:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503978240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760962.238510:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503978368:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760962.238522:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503978496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760962.238534:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503978624:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760962.878214:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760963.902177:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760964.926205:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789760965.310366:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789760965.310407:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789760965.310424:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789760965.310467:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789760965.310478:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789760965.310491:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789760965.310503:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789760965.310516:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789760965.310541:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760962/real 1789760965] req@ffff9018c8b9bb80 x1876699503978240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760965.310584:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699503978240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760965.310606:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760965.310612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760965.310617:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503978240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760965.310628:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760962/real 1789760965] req@ffff9018c8b98700 x1876699503978368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760965.310642:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699503978368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760965.310651:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760965.310656:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760965.310660:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503978368:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760965.310668:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760962/real 1789760965] req@ffff9018c8b98a80 x1876699503978496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760965.310680:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699503978496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760965.310688:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760965.310691:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760965.310694:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503978496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760965.310702:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760962/real 1789760965] req@ffff9018c8b98e00 x1876699503978624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760965.310716:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699503978624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760965.310725:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760965.310729:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760965.310734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503978624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760965.310742:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760962/real 1789760965] req@ffff9018c8b99880 x1876699503977984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760965.310757:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699503977984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760965.310766:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760965.310770:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760965.310774:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503977984:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760965.310783:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760962/real 1789760965] req@ffff9018c8b99f80 x1876699503978112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760965.310798:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699503978112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760965.310807:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760965.310811:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760965.310815:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503978112:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760965.950225:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760966.974225:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760967.358211:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760967.358221:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760967.358225:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760967.358235:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 893 00000100:00080000:2.0:1789760967.358246:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760967.358253:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760967.358310:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699503978752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760967.358346:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760967.358351:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760967.358353:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760967.358358:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 893 00000100:00080000:2.0:1789760967.358363:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760967.358368:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760967.358381:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760967.358397:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699503978880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760967.358410:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760967.358414:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760967.358417:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760967.358418:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503978752:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789760967.358423:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 893 00000100:00080000:2.0:1789760967.358431:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760967.358435:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760967.358441:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760967.358456:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699503979008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760967.358467:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760967.358472:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760967.358473:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760967.358478:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 893 00000100:00080000:2.0:1789760967.358498:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760967.358503:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760967.358509:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789760967.358522:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760967.358525:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699503979136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760967.358544:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760967.358548:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760967.358550:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760967.358554:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 893 00000100:00080000:2.0:1789760967.358559:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760967.358563:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760967.358569:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760967.358582:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699503979264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760967.358593:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760967.358594:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760967/real 1789760967] req@ffff9018d0f24a80 x1876699503978752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760967.358601:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760967.358607:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760967.358616:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 893 00000100:00080000:2.0:1789760967.358621:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760967.358626:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789760967.358630:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699503978752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789760967.358634:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789760967.358652:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789760967.358654:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699503979392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760967.358662:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760967.358672:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503978752:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760967.358712:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503978880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760967.358741:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760967.358757:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760967/real 1789760967] req@ffff9018d0f25180 x1876699503978880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760967.358773:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699503978880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760967.358782:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760967.358785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760967.358795:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503978880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760967.358802:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503979008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760967.358824:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760967.358836:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760967/real 1789760967] req@ffff9018d0f27800 x1876699503979008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760967.358849:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699503979008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760967.358863:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760967.358866:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760967.358870:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503979008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760967.358876:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503979136:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760967.358898:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760967.358911:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760967/real 1789760967] req@ffff9018d0f26d80 x1876699503979136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760967.358923:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699503979136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760967.358932:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760967.358936:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760967.358940:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503979136:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760967.358947:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503979264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760967.358970:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760967.358983:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760967/real 1789760967] req@ffff9018d0f25500 x1876699503979264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760967.358993:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699503979264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760967.358999:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760967.359001:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760967.359004:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503979264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760967.359008:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503979392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760967.359023:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760967.359150:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760967/real 1789760967] req@ffff9018d0f24000 x1876699503979392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760967.359163:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699503979392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760967.359170:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760967.359173:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760967.359177:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503979392:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789760967.998327:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760969.022390:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760970.046309:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760971.070188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760972.094181:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760972.478187:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760972.478195:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760972.478199:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760972.478206:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 899 00000100:00080000:1.0:1789760972.478215:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760972.478221:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789760972.478267:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699503979520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760972.478291:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760972.478298:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760972.478300:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760972.478303:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 899 00000100:00080000:1.0:1789760972.478305:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760972.478308:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760972.478316:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760972.478325:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699503979648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760972.478341:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760972.478344:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760972.478345:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760972.478351:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 899 00000100:00100000:3.0:1789760972.478351:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503979520:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789760972.478357:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760972.478364:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760972.478367:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760972.478375:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699503979776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760972.478381:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760972.478383:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760972.478385:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760972.478387:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 899 00000100:00080000:1.0:1789760972.478389:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760972.478392:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760972.478395:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789760972.478402:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699503979904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760972.478409:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760972.478411:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760972.478412:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760972.478417:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 899 00000800:00000100:3.0:1789760972.478419:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789760972.478421:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760972.478425:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760972.478428:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760972.478436:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699503980032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760972.478442:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760972.478445:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760972.478446:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760972.478448:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 899 00000100:00080000:1.0:1789760972.478451:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760972.478453:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760972.478456:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760972.478462:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699503980160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760972.478474:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760972/real 1789760972] req@ffff9018c66a5180 x1876699503979520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760972.478496:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699503979520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760972.478508:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760972.478512:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760972.478517:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503979520:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760972.478549:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503979648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760972.478571:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760972.478583:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760972/real 1789760972] req@ffff9018c66a6680 x1876699503979648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760972.478607:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699503979648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760972.478616:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760972.478620:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760972.478624:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503979648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760972.478630:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503979776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760972.478658:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760972.478668:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760972/real 1789760972] req@ffff9018c66a4700 x1876699503979776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760972.478681:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699503979776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760972.478690:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760972.478693:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760972.478697:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503979776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760972.478703:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503979904:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760972.478719:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760972.478729:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760972/real 1789760972] req@ffff9018c66a6300 x1876699503979904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760972.478742:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699503979904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760972.478751:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760972.478754:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760972.478758:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503979904:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760972.478763:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503980032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760972.478779:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760972.478789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760972/real 1789760972] req@ffff9018c66a4e00 x1876699503980032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760972.478801:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699503980032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760972.478810:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760972.478813:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760972.478817:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503980032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760972.478823:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503980160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760972.478838:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760972.478847:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760972/real 1789760972] req@ffff9018c66a5500 x1876699503980160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760972.478859:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699503980160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760972.478874:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760972.478878:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760972.478881:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503980160:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760973.118350:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760974.142258:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760975.166210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760976.190276:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760977.214189:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760977.598189:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760977.598223:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760977.598230:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760977.598248:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 904 00000100:00080000:1.0:1789760977.598275:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760977.598288:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789760977.598345:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699503980288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760977.598396:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760977.598402:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760977.598405:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760977.598412:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 904 00000100:00080000:1.0:1789760977.598419:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760977.598427:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760977.598450:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503980288:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789760977.598455:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760977.598480:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699503980416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760977.598498:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760977.598504:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760977.598507:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760977.598514:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 904 00000100:00080000:1.0:1789760977.598521:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760977.598528:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789760977.598535:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:1.0:1789760977.598536:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760977.598556:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699503980544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760977.598577:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789760977.598582:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760977/real 1789760977] req@ffff9018c66a7800 x1876699503980288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760977.598585:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760977.598591:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760977.598606:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 904 00000100:00080000:1.0:1789760977.598615:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789760977.598621:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699503980288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760977.598625:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760977.598640:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1789760977.598641:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760977.598649:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760977.598654:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503980288:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789760977.598665:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699503980672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760977.598683:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503980416:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789760977.598703:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760977.598709:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760977.598712:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760977.598718:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 904 00000100:00080000:1.0:1789760977.598724:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789760977.598730:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789760977.598733:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760977.598742:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760977.598749:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760977/real 1789760977] req@ffff9018c66a7b80 x1876699503980416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760977.598761:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699503980800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760977.598767:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699503980416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760977.598796:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760977.598799:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760977.598802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789760977.598811:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789760977.598812:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503980416:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789760977.598818:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789760977.598824:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503980544:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789760977.598830:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 904 00000100:00080000:1.0:1789760977.598838:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760977.598844:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760977.598852:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789760977.598864:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789760977.598871:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699503980928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760977.598887:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760977/real 1789760977] req@ffff9018c66a4000 x1876699503980544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760977.598908:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699503980544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760977.598928:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760977.598932:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760977.598936:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503980544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760977.598958:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503980672:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789760977.598990:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760977.599008:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760977/real 1789760977] req@ffff9018c66a5f80 x1876699503980672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760977.599023:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699503980672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760977.599074:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760977.599081:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760977.599088:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503980672:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789760977.599099:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503980800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760977.599133:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760977.599159:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760977/real 1789760977] req@ffff9018c66a6d80 x1876699503980800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760977.599179:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699503980800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760977.599190:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760977.599195:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760977.599219:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503980800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760977.599228:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503980928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760977.599256:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760977.599273:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760977/real 1789760977] req@ffff9018c66a5c00 x1876699503980928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760977.599290:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699503980928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760977.599300:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760977.599305:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760977.599309:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503980928:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760978.238334:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760979.262239:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760980.286243:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760981.310261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760982.334217:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760982.718151:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760982.718173:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760982.718177:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760982.718199:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 909 00000100:00080000:1.0:1789760982.718210:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760982.718217:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789760982.718254:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699503981056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760982.718292:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760982.718296:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760982.718298:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760982.718302:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 909 00000100:00080000:1.0:1789760982.718306:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760982.718310:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760982.718336:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760982.718354:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699503981184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760982.718367:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760982.718373:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760982.718375:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760982.718385:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 909 00000100:00080000:1.0:1789760982.718390:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789760982.718393:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503981056:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789760982.718395:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760982.718402:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760982.718432:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699503981312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760982.718442:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760982.718445:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760982.718446:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760982.718451:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 909 00000100:00080000:1.0:1789760982.718453:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760982.718456:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760982.718464:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789760982.718505:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699503981440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760982.718516:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760982.718520:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760982.718522:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760982.718531:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 909 00000800:00000100:2.0:1789760982.718531:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789760982.718543:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760982.718548:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760982.718555:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760982.718590:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699503981568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760982.718600:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760982.718614:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760982.718616:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760982.718620:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 909 00000100:00080000:1.0:1789760982.718623:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760982.718628:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760982.718650:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760982/real 1789760982] req@ffff9018c66a5c00 x1876699503981056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789760982.718651:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760982.718673:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699503981696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760982.718688:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699503981056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760982.718704:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760982.718709:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760982.718714:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503981056:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789760982.718746:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503981184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760982.718772:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760982.718785:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760982/real 1789760982] req@ffff9018c66a6d80 x1876699503981184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760982.718797:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699503981184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760982.718805:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760982.718808:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760982.718811:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503981184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760982.718816:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503981312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760982.718831:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760982.718839:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760982/real 1789760982] req@ffff9018c66a5f80 x1876699503981312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760982.718849:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699503981312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760982.718856:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760982.718858:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760982.718861:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503981312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760982.718883:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503981440:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789760982.718897:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760982.718904:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760982/real 1789760982] req@ffff9018c66a4000 x1876699503981440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760982.718911:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699503981440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760982.718918:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760982.718921:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760982.718923:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503981440:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789760982.718928:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503981568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760982.718939:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760982.718946:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760982/real 1789760982] req@ffff9018c66a7b80 x1876699503981568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760982.718953:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699503981568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760982.718959:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760982.718962:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760982.718964:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503981568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760982.718968:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503981696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760982.718998:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760982.719006:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760982/real 1789760982] req@ffff9018c66a7800 x1876699503981696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760982.719016:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699503981696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760982.719023:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760982.719026:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760982.719029:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503981696:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789760983.358222:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760984.382288:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760985.406281:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760986.430257:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760987.454387:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760987.838216:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760987.838224:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760987.838227:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760987.838236:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 914 00000100:00080000:0.0:1789760987.838245:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760987.838250:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789760987.838283:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699503981824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760987.838303:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760987.838306:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760987.838307:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760987.838310:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 914 00000100:00080000:0.0:1789760987.838312:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760987.838315:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760987.838322:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760987.838331:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699503981952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760987.838340:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760987.838342:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760987.838344:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760987.838351:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 914 00000100:00080000:0.0:1789760987.838353:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760987.838357:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760987.838363:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760987.838374:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699503982080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760987.838390:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503981824:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789760987.838393:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760987.838400:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760987.838402:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760987.838409:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 914 00000100:00080000:0.0:1789760987.838416:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760987.838420:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760987.838424:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789760987.838439:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699503982208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760987.838451:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760987.838454:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760987.838457:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760987.838461:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 914 00000100:00080000:0.0:1789760987.838466:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760987.838471:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789760987.838476:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760987.838486:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699503982336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760987.838492:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760987.838495:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789760987.838496:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760987.838498:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 914 00000100:00080000:0.0:1789760987.838501:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760987.838507:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789760987.838515:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789760987.838517:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760987.838528:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699503982464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760987.838593:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760987/real 1789760987] req@ffff9018c8b9b480 x1876699503981824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760987.838617:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699503981824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760987.838640:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760987.838646:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760987.838651:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503981824:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789760987.838662:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503981952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760987.838692:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760987.838717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760987/real 1789760987] req@ffff9018c8b9b800 x1876699503981952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760987.838732:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699503981952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760987.838742:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760987.838746:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760987.838750:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503981952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760987.838791:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503982080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760987.838816:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760987.838830:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760987/real 1789760987] req@ffff9018c8b9aa00 x1876699503982080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760987.838853:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699503982080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760987.838863:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760987.838866:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760987.838870:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503982080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760987.838878:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503982208:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789760987.838902:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760987.838916:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760987/real 1789760987] req@ffff9018c8b98000 x1876699503982208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760987.838938:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699503982208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760987.838948:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760987.838952:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760987.838956:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503982208:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789760987.838963:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503982336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760987.838988:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760987.839001:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760987/real 1789760987] req@ffff9018c8b9a680 x1876699503982336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760987.839014:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699503982336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760987.839024:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760987.839028:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760987.839032:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503982336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760987.839068:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503982464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760987.839090:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760987.839102:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760987/real 1789760987] req@ffff9018c8b9b100 x1876699503982464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760987.839115:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699503982464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760987.839140:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760987.839144:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760987.839148:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503982464:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760988.478235:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760989.502283:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760990.526233:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760991.550241:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760992.574241:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760992.958251:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760992.958261:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760992.958266:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760992.958278:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 919 00000100:00080000:1.0:1789760992.958290:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760992.958298:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789760992.958345:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699503982592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760992.958371:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760992.958375:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760992.958376:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760992.958380:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 919 00000100:00080000:1.0:1789760992.958383:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760992.958387:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760992.958398:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760992.958410:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699503982720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760992.958418:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760992.958420:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760992.958422:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760992.958426:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 919 00000100:00080000:1.0:1789760992.958428:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760992.958431:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760992.958436:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760992.958447:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699503982848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760992.958449:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503982592:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789760992.958475:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760992.958478:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760992.958479:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760992.958482:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 919 00000100:00080000:1.0:1789760992.958485:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760992.958488:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760992.958495:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789760992.958505:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699503982976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760992.958518:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760992.958522:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760992.958524:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760992.958527:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 919 00000100:00080000:1.0:1789760992.958530:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760992.958533:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760992.958536:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789760992.958537:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789760992.958556:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699503983104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760992.958564:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760992.958567:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789760992.958569:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760992.958572:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 919 00000100:00080000:1.0:1789760992.958576:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760992.958578:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789760992.958582:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760992.958588:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699503983232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760992.958620:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760992/real 1789760992] req@ffff9018c66a5500 x1876699503982592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760992.958645:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699503982592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760992.958655:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760992.958674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760992.958678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503982592:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789760992.958686:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503982720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760992.958709:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760992.958721:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760992/real 1789760992] req@ffff9018c66a4e00 x1876699503982720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760992.958731:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699503982720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760992.958737:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760992.958740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760992.958742:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503982720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760992.958794:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503982848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760992.958816:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760992.958844:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760992/real 1789760992] req@ffff9018c66a6300 x1876699503982848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760992.958854:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699503982848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760992.958859:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760992.958862:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760992.958864:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503982848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760992.958869:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503982976:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789760992.958901:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760992.958914:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760992/real 1789760992] req@ffff9018c66a4700 x1876699503982976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760992.958926:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699503982976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760992.958932:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760992.958934:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760992.958936:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503982976:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789760992.958958:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503983104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760992.958976:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760992.958983:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760992/real 1789760992] req@ffff9018c66a6680 x1876699503983104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760992.958991:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699503983104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760992.958996:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760992.958998:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760992.959000:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503983104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789760992.959004:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503983232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789760992.959016:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760992.959022:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760992/real 1789760992] req@ffff9018c66a5180 x1876699503983232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760992.959030:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699503983232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760992.959036:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760992.959038:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789760992.959040:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503983232:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789760993.598339:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760994.622197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760995.646320:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760996.670201:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760997.694221:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760998.078188:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760998.078199:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760998.078204:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760998.078215:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 924 00000100:00080000:2.0:1789760998.078229:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760998.078236:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789760998.078280:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699503983360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760998.078309:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760998.078312:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760998.078313:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760998.078317:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 924 00000100:00080000:2.0:1789760998.078319:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760998.078322:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760998.078332:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760998.078342:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699503983488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760998.078351:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760998.078353:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760998.078356:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760998.078361:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 924 00000100:00100000:3.0:1789760998.078363:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503983360:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789760998.078364:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760998.078371:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760998.078376:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760998.078384:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699503983616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760998.078391:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760998.078394:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760998.078395:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760998.078397:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 924 00000100:00080000:2.0:1789760998.078400:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760998.078402:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760998.078406:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789760998.078414:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699503983744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760998.078420:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760998.078423:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760998.078425:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760998.078427:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 924 00000100:00080000:2.0:1789760998.078430:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760998.078433:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760998.078436:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789760998.078438:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789760998.078451:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699503983872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760998.078459:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760998.078462:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789760998.078463:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760998.078466:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 924 00000100:00080000:2.0:1789760998.078467:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760998.078470:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789760998.078473:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760998.078481:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699503984000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760998.078489:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760998/real 1789760998] req@ffff9018d0f24e00 x1876699503983360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760998.078507:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699503983360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760998.078522:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760998.078526:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760998.078530:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503983360:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789760998.078560:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503983488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760998.078586:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760998.078599:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760998/real 1789760998] req@ffff9018d0f26680 x1876699503983488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760998.078613:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699503983488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760998.078622:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760998.078626:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760998.078630:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503983488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760998.078637:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503983616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760998.078658:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760998.078667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760998/real 1789760998] req@ffff9018d0f24700 x1876699503983616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760998.078676:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699503983616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760998.078682:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760998.078684:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760998.078686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503983616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760998.078691:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503983744:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789760998.078703:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760998.078710:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760998/real 1789760998] req@ffff9018d0f27480 x1876699503983744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760998.078718:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699503983744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760998.078723:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760998.078725:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760998.078727:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503983744:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789760998.078730:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503983872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760998.078741:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760998.078747:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760998/real 1789760998] req@ffff9018d0f24000 x1876699503983872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760998.078754:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699503983872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760998.078759:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760998.078761:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760998.078763:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503983872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789760998.078766:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503984000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789760998.078776:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789760998.078781:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760998/real 1789760998] req@ffff9018d0f25500 x1876699503984000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760998.078789:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699503984000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760998.078796:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760998.078799:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789760998.078803:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503984000:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789760998.718208:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760999.742236:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761000.766288:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761001.790150:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761002.814191:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761003.198108:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761003.198118:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761003.198122:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761003.198132:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 929 00000100:00080000:2.0:1789761003.198143:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761003.198151:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761003.198189:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699503984128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761003.198219:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761003.198223:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761003.198225:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761003.198230:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 929 00000100:00080000:2.0:1789761003.198235:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761003.198241:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761003.198251:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761003.198254:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503984128:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761003.198269:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699503984256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761003.198284:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761003.198288:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761003.198291:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761003.198295:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 929 00000100:00080000:2.0:1789761003.198317:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761003.198322:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789761003.198327:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789761003.198337:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761003.198358:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699503984384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761003.198367:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761003/real 1789761003] req@ffff9018d0f26d80 x1876699503984128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761003.198375:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761003.198386:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761003.198390:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761003.198396:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 929 00000100:00080000:2.0:1789761003.198401:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789761003.198403:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699503984128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761003.198409:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789761003.198416:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789761003.198419:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1789761003.198422:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761003.198429:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503984128:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761003.198441:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699503984512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761003.198453:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503984256:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761003.198457:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761003.198464:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761003.198466:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761003.198472:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 929 00000800:00000100:3.0:1789761003.198477:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761003.198480:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761003.198489:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761003.198492:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761003/real 1789761003] req@ffff9018d0f27800 x1876699503984256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761003.198499:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761003.198514:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699503984256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761003.198522:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699503984640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761003.198526:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761003.198552:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789761003.198560:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761003.198560:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503984256:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761003.198570:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789761003.198575:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503984384:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761003.198576:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761003.198587:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 929 00000100:00080000:2.0:1789761003.198593:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761003.198598:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789761003.198603:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789761003.198606:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761003.198617:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761003/real 1789761003] req@ffff9018d0f25180 x1876699503984384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761003.198625:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699503984768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761003.198638:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699503984384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761003.198650:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761003.198655:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761003.198659:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503984384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761003.198672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503984512:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761003.198689:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761003.198696:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761003/real 1789761003] req@ffff9018d0f24a80 x1876699503984512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761003.198704:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699503984512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761003.198714:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761003.198716:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761003.198719:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503984512:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761003.198722:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503984640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761003.198732:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761003.198738:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761003/real 1789761003] req@ffff9018d0f25f80 x1876699503984640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761003.198745:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699503984640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761003.198751:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761003.198753:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761003.198754:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503984640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761003.198758:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503984768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761003.198767:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761003.198773:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761003/real 1789761003] req@ffff9018d0f25c00 x1876699503984768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761003.198780:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699503984768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761003.198785:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761003.198787:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761003.198790:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503984768:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761003.838137:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761004.862265:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761005.886158:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761006.910158:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761007.934209:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761008.318117:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761008.318128:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761008.318133:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761008.318145:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 934 00000100:00080000:3.0:1789761008.318160:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761008.318170:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761008.318218:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699503984896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761008.318245:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761008.318248:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761008.318250:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761008.318254:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 934 00000100:00080000:3.0:1789761008.318257:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761008.318262:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761008.318272:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761008.318277:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503984896:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761008.318285:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699503985024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761008.318294:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761008.318297:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761008.318299:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761008.318303:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 934 00000100:00080000:3.0:1789761008.318305:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761008.318308:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761008.318312:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761008.318320:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c380 x1876699503985152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761008.318327:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761008.318330:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761008.318331:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761008.318334:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 934 00000100:00080000:3.0:1789761008.318339:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761008.318341:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761008.318345:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789761008.318350:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761008.318355:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699503985280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761008.318371:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761008.318374:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761008.318375:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761008.318378:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 934 00000100:00080000:3.0:1789761008.318382:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761008.318384:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761008.318388:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761008.318390:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761008/real 1789761008] req@ffff9018d070d180 x1876699503984896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761008.318399:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699503985408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761008.318409:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761008.318412:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699503984896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761008.318413:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761008.318416:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761008.318423:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 934 00000100:00080000:3.0:1789761008.318427:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761008.318433:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761008.318439:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789761008.318443:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761008.318449:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761008.318456:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503984896:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761008.318457:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699503985536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761008.318491:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503985024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761008.318517:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761008.318528:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761008/real 1789761008] req@ffff9018d070e300 x1876699503985024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761008.318537:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699503985024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761008.318543:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761008.318546:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761008.318548:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503985024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761008.318552:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503985152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761008.318565:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761008.318571:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761008/real 1789761008] req@ffff9018d070c380 x1876699503985152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761008.318579:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c380 x1876699503985152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761008.318584:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761008.318586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761008.318588:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503985152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761008.318592:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503985280:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761008.318603:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761008.318609:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761008/real 1789761008] req@ffff9018d070dc00 x1876699503985280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761008.318616:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699503985280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761008.318631:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761008.318633:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761008.318635:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503985280:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761008.318639:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503985408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761008.318651:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761008.318658:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761008/real 1789761008] req@ffff9018d070c000 x1876699503985408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761008.318666:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699503985408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761008.318671:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761008.318673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761008.318675:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503985408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761008.318678:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503985536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761008.318689:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761008.318695:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761008/real 1789761008] req@ffff9018d070fb80 x1876699503985536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761008.318703:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699503985536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761008.318708:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761008.318710:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761008.318712:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503985536:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761008.958206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761009.982185:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761011.006209:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761012.030241:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761013.054181:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761013.438158:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761013.438163:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761013.438167:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761013.438175:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 939 00000100:00080000:3.0:1789761013.438184:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761013.438190:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761013.438218:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699503985664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761013.438240:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761013.438243:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761013.438245:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761013.438248:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 939 00000100:00080000:3.0:1789761013.438251:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761013.438260:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761013.438270:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761013.438286:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699503985792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761013.438296:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503985664:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761013.438312:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761013.438316:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761013.438319:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761013.438324:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 939 00000100:00080000:3.0:1789761013.438329:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761013.438334:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761013.438340:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761013.438356:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699503985920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761013.438361:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761013.438371:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761013.438374:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761013.438375:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761013.438379:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 939 00000100:00080000:3.0:1789761013.438384:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761013.438400:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761013/real 1789761013] req@ffff9018d070d880 x1876699503985664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761013.438419:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699503985664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761013.438428:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761013.438431:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761013.438434:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503985664:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761013.438452:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761013.438456:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503985792:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789761013.438462:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761013.438479:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699503986048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761013.438480:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761013.438493:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761013.438494:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761013/real 1789761013] req@ffff9018d070ce00 x1876699503985792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761013.438500:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761013.438505:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761013.438513:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 939 00000100:00100000:1.0:1789761013.438517:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699503985792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761013.438519:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761013.438528:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789761013.438529:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761013.438537:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761013.438540:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503985792:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789761013.438540:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761013.438550:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503985920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761013.438559:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699503986176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761013.438566:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761013.438577:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761013/real 1789761013] req@ffff9018d070ca80 x1876699503985920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761013.438577:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761013.438588:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761013.438592:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761013.438596:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699503985920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761013.438599:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 939 00000100:00080000:3.0:1789761013.438607:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761013.438613:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789761013.438622:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789761013.438622:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761013.438629:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761013.438632:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503985920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761013.438637:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699503986304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761013.438645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503986048:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761013.438661:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761013.438669:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761013/real 1789761013] req@ffff9018d070df80 x1876699503986048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761013.438677:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699503986048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761013.438686:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761013.438689:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761013.438693:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503986048:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761013.438699:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503986176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761013.438719:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761013.438734:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761013/real 1789761013] req@ffff9018d070ed80 x1876699503986176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761013.438749:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699503986176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761013.438757:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761013.438761:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761013.438767:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503986176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761013.438790:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503986304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761013.438807:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761013.438815:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761013/real 1789761013] req@ffff9018d070f480 x1876699503986304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761013.438824:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699503986304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761013.438830:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761013.438832:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761013.438834:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503986304:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761014.078151:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761015.102203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761016.126221:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761017.150162:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761018.174214:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761018.558152:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761018.558163:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761018.558166:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761018.558177:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 945 00000100:00080000:3.0:1789761018.558188:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761018.558195:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761018.558239:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699503986432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761018.558267:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761018.558271:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761018.558272:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761018.558276:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 945 00000100:00080000:3.0:1789761018.558278:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761018.558280:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761018.558290:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761018.558299:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699503986560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761018.558306:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761018.558314:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761018.558316:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761018.558320:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 945 00000100:00080000:3.0:1789761018.558322:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761018.558326:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761018.558330:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503986432:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789761018.558331:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761018.558345:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699503986688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761018.558358:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761018.558362:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761018.558364:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761018.558369:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 945 00000100:00080000:3.0:1789761018.558381:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761018.558386:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761018.558393:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761018.558406:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699503986816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761018.558410:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761018.558420:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761018.558423:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761018.558425:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761018.558429:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 945 00000100:00080000:3.0:1789761018.558433:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761018.558436:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761018.558439:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761018.558450:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699503986944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761018.558459:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761018/real 1789761018] req@ffff9018d070f480 x1876699503986432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761018.558471:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761018.558477:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761018.558479:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761018.558485:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 945 00000100:00080000:3.0:1789761018.558490:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761018.558498:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699503986432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761018.558498:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761018.558504:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761018.558516:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761018.558520:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699503987072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761018.558521:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761018.558532:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503986432:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761018.558544:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503986560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761018.558573:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761018.558585:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761018/real 1789761018] req@ffff9018d070ed80 x1876699503986560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761018.558594:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699503986560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761018.558601:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761018.558603:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761018.558606:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503986560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761018.558629:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503986688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761018.558642:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761018.558649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761018/real 1789761018] req@ffff9018d070df80 x1876699503986688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761018.558657:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699503986688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761018.558663:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761018.558665:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761018.558667:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503986688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761018.558671:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503986816:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761018.558682:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761018.558688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761018/real 1789761018] req@ffff9018d070ca80 x1876699503986816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761018.558695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699503986816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761018.558702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761018.558704:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761018.558706:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503986816:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761018.558709:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503986944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761018.558719:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761018.558725:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761018/real 1789761018] req@ffff9018d070ce00 x1876699503986944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761018.558737:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699503986944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761018.558745:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761018.558748:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761018.558752:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503986944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761018.558759:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503987072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761018.558780:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761018.558789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761018/real 1789761018] req@ffff9018d070d880 x1876699503987072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761018.558797:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699503987072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761018.558803:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761018.558805:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761018.558808:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503987072:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761019.198188:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761020.222207:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761021.246216:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761022.270161:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761023.294226:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761023.678093:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761023.678104:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761023.678108:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761023.678118:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 950 00000100:00080000:3.0:1789761023.678129:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761023.678137:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761023.678188:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699503987200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761023.678221:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761023.678226:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761023.678228:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761023.678233:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 950 00000100:00080000:3.0:1789761023.678238:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761023.678244:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761023.678256:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761023.678284:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503987200:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761023.678284:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699503987328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761023.678304:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761023.678308:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761023.678310:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761023.678315:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 950 00000100:00080000:3.0:1789761023.678319:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761023.678323:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761023.678329:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761023.678344:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699503987456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761023.678355:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761023.678359:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761023.678362:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761023.678366:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 950 00000100:00080000:3.0:1789761023.678371:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761023.678375:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761023.678381:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789761023.678386:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761023.678397:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c380 x1876699503987584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761023.678406:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761023.678409:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761023.678411:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761023.678414:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 950 00000100:00080000:3.0:1789761023.678418:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761023.678421:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761023.678425:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761023.678434:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699503987712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761023.678438:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761023/real 1789761023] req@ffff9018d070fb80 x1876699503987200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761023.678448:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761023.678455:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761023.678457:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761023.678461:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 950 00000100:00080000:3.0:1789761023.678464:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761023.678467:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699503987200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761023.678471:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761023.678478:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761023.678484:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761023.678489:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699503987840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761023.678490:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761023.678498:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503987200:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761023.678533:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503987328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761023.678561:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761023.678574:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761023/real 1789761023] req@ffff9018d070c000 x1876699503987328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761023.678583:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699503987328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761023.678590:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761023.678592:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761023.678595:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503987328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761023.678599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503987456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761023.678612:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761023.678620:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761023/real 1789761023] req@ffff9018d070dc00 x1876699503987456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761023.678628:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699503987456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761023.678633:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761023.678635:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761023.678637:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503987456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761023.678641:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503987584:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761023.678652:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761023.678659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761023/real 1789761023] req@ffff9018d070c380 x1876699503987584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761023.678666:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c380 x1876699503987584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761023.678672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761023.678674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761023.678675:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503987584:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761023.678678:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503987712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761023.678689:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761023.678694:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761023/real 1789761023] req@ffff9018d070e300 x1876699503987712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761023.678702:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699503987712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761023.678707:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761023.678710:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761023.678712:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503987712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761023.678715:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503987840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761023.678727:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761023.678732:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761023/real 1789761023] req@ffff9018d070d180 x1876699503987840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761023.678739:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699503987840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761023.678744:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761023.678746:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761023.678748:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503987840:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761024.318256:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761025.342164:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761026.366290:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761027.390237:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761028.414262:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761028.798140:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761028.798152:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761028.798156:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761028.798167:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 955 00000100:00080000:2.0:1789761028.798179:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761028.798185:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761028.798241:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699503987968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761028.798276:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761028.798281:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761028.798283:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761028.798288:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 955 00000100:00080000:2.0:1789761028.798293:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761028.798299:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761028.798312:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761028.798319:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503987968:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761028.798327:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699503988096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761028.798343:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761028.798345:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761028.798347:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761028.798350:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 955 00000100:00080000:2.0:1789761028.798353:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761028.798357:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761028.798360:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761028.798376:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699503988224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761028.798387:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761028.798392:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761028.798394:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761028.798399:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 955 00000100:00080000:2.0:1789761028.798404:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761028.798409:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761028.798415:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761028.798430:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699503988352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761028.798440:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503988096:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761028.798440:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761028.798447:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761028.798448:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761028.798453:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 955 00000100:00080000:2.0:1789761028.798458:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761028.798461:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761028.798466:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761028.798470:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503988224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761028.798487:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699503988480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761028.798497:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503988352:192.168.203.158@tcp:8:kworker.0 00000100:00080000:2.0:1789761028.798505:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761028.798511:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761028.798513:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761028.798520:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 955 00000100:00100000:1.0:1789761028.798521:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503988480:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761028.798529:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761028.798537:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761028.798544:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761028.798559:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699503988608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761028.798599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503988608:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761029.438188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761030.462192:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761031.486183:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761031.870341:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:0.0:1789761031.870397:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761031.870418:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789761031.870465:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789761031.870476:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789761031.870486:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789761031.870498:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789761031.870511:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789761031.870535:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761028/real 1789761031] req@ffff9018d0f25f80 x1876699503988608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761031.870570:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699503988608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761031.870584:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761031.870590:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761031.870595:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503988608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761031.870604:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761028/real 1789761031] req@ffff9018d0f25c00 x1876699503988480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761031.870619:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699503988480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761031.870628:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761031.870632:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761031.870636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503988480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761031.870644:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761028/real 1789761031] req@ffff9018d0f26a00 x1876699503988352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761031.870656:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699503988352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761031.870665:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761031.870669:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761031.870673:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503988352:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761031.870682:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761028/real 1789761031] req@ffff9018d0f27100 x1876699503988096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761031.870696:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699503988096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761031.870704:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761031.870708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761031.870711:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503988096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761031.870719:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761028/real 1789761031] req@ffff9018d0f26300 x1876699503988224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761031.870731:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699503988224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761031.870739:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761031.870743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761031.870747:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503988224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761031.870754:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761028/real 1789761031] req@ffff9018d0f24380 x1876699503987968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761031.870767:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699503987968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761031.870776:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761031.870779:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761031.870783:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503987968:192.168.203.158@tcp:250:kworker.0 00010000:00010000:0.0:1789761032.510230:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761033.534215:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761033.918160:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761033.918187:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761033.918192:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761033.918207:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 960 00000100:00080000:2.0:1789761033.918221:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761033.918230:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761033.918285:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699503988736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761033.918320:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761033.918327:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761033.918329:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761033.918334:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 960 00000100:00080000:2.0:1789761033.918337:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761033.918341:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761033.918352:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761033.918364:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699503988864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761033.918374:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761033.918376:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761033.918378:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761033.918383:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 960 00000100:00100000:1.0:1789761033.918385:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503988736:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761033.918387:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761033.918396:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761033.918402:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761033.918416:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699503988992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761033.918426:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761033.918430:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761033.918432:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761033.918437:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 960 00000100:00080000:2.0:1789761033.918448:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761033.918453:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761033.918459:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789761033.918476:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761033.918476:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699503989120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761033.918490:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761033.918494:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761033.918497:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761033.918501:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 960 00000100:00080000:2.0:1789761033.918508:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761033.918512:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761033.918517:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761033.918536:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761033/real 1789761033] req@ffff9018d0f24a80 x1876699503988736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761033.918537:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699503989248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761033.918573:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761033.918577:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761033.918579:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761033.918584:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699503988736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761033.918587:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 960 00000100:00080000:2.0:1789761033.918595:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761033.918602:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789761033.918604:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789761033.918608:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761033.918612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761033.918619:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503988736:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761033.918625:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699503989376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761033.918655:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503988864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761033.918683:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761033.918698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761033/real 1789761033] req@ffff9018d0f25180 x1876699503988864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761033.918712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699503988864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761033.918721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761033.918724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761033.918728:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503988864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761033.918734:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503988992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761033.918754:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761033.918766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761033/real 1789761033] req@ffff9018d0f27800 x1876699503988992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761033.918793:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699503988992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761033.918802:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761033.918805:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761033.918808:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503988992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761033.918815:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503989120:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761033.918836:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761033.918848:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761033/real 1789761033] req@ffff9018d0f26d80 x1876699503989120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761033.918860:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699503989120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761033.918867:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761033.918871:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761033.918873:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503989120:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761033.918880:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503989248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761033.918901:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761033.918912:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761033/real 1789761033] req@ffff9018d0f25500 x1876699503989248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761033.918925:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699503989248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761033.918940:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761033.918943:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761033.918946:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503989248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761033.918953:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503989376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761033.918973:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761033.918985:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761033/real 1789761033] req@ffff9018d0f24000 x1876699503989376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761033.918998:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699503989376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761033.919005:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761033.919007:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761033.919009:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503989376:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761034.558254:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761035.582159:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761036.606271:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761037.630181:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761038.654217:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761039.038154:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761039.038164:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761039.038169:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761039.038181:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 965 00000100:00080000:3.0:1789761039.038194:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761039.038201:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761039.038246:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699503989504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761039.038275:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761039.038278:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761039.038279:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761039.038283:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 965 00000100:00080000:3.0:1789761039.038285:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761039.038287:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761039.038305:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761039.038319:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699503989632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761039.038327:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503989504:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761039.038340:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761039.038345:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761039.038346:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761039.038352:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 965 00000100:00080000:3.0:1789761039.038357:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761039.038362:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761039.038368:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761039.038379:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699503989760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761039.038392:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761039.038394:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761039.038396:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761039.038398:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 965 00000100:00080000:3.0:1789761039.038402:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761039.038403:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761039.038407:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761039.038412:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761039.038421:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699503989888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761039.038429:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761039.038432:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761039.038433:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761039.038436:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 965 00000100:00080000:3.0:1789761039.038438:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761039.038441:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761039.038444:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761039.038450:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761039/real 1789761039] req@ffff9018d070f100 x1876699503989504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761039.038455:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699503990016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761039.038470:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761039.038473:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761039.038474:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761039.038476:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 965 00000100:00080000:3.0:1789761039.038480:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761039.038482:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761039.038485:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761039.038487:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699503989504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761039.038494:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699503990144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761039.038500:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761039.038506:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761039.038511:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503989504:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761039.038533:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503989632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761039.038551:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761039.038560:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761039/real 1789761039] req@ffff9018d070e680 x1876699503989632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761039.038569:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699503989632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761039.038575:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761039.038577:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761039.038579:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503989632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761039.038583:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503989760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761039.038594:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761039.038600:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761039/real 1789761039] req@ffff9018d070d500 x1876699503989760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761039.038607:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699503989760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761039.038613:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761039.038614:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761039.038616:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503989760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761039.038620:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503989888:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761039.038634:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761039.038645:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761039/real 1789761039] req@ffff9018d070f800 x1876699503989888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761039.038658:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699503989888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761039.038666:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761039.038669:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761039.038673:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503989888:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761039.038679:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503990016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761039.038701:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761039.038715:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761039/real 1789761039] req@ffff9018d070d180 x1876699503990016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761039.038727:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699503990016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761039.038736:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761039.038755:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761039.038759:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503990016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761039.038765:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503990144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761039.038785:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761039.038795:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761039/real 1789761039] req@ffff9018d070e300 x1876699503990144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761039.038807:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699503990144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761039.038816:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761039.038820:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761039.038823:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503990144:192.168.203.158@tcp:38:kworker.0 00010000:00010000:1.0:1789761039.678176:0:254:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1789761039.678229:0:6165:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1789761039.678242:0:6165:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9018c54e9800 lock: ffff9018c0031200/0xeac59f702d4e6ee0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xfa2da65b083d07df expref: -99 pid: 12577 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789761039.678251:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1789761039.678287:0:6165:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6:0x0]: losing layout lock 00010000:00010000:3.0:1789761039.678317:0:6165:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9018c54e9800 lock: ffff9018c0030400/0xeac59f702d4e6ed9 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xfa2da65b083d07c3 expref: -99 pid: 12577 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789761039.678328:0:6165:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9018c54e9800 lock: ffff9018d1806600/0xeac59f702d4e6ec4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xfa2da65b083d0761 expref: -99 pid: 12570 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789761039.678377:0:6165:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9018c54e9800 lock: ffff9018c0031200/0xeac59f702d4e6ee0 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xfa2da65b083d07df expref: -99 pid: 12577 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789761039.678383:0:6165:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9018c54e9800 lock: ffff9018c0030400/0xeac59f702d4e6ed9 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xfa2da65b083d07c3 expref: -99 pid: 12577 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789761039.678388:0:6165:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9018c54e9800 lock: ffff9018d1806600/0xeac59f702d4e6ec4 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xfa2da65b083d0761 expref: -99 pid: 12570 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789761039.678391:0:6165:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1789761039.678420:0:6165:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9018c54e9800 lock: ffff9018c0031200/0xeac59f702d4e6ee0 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xfa2da65b083d07df expref: -99 pid: 12577 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789761039.678448:0:6165:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9018c54e9800 lock: ffff9018c0030400/0xeac59f702d4e6ed9 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xfa2da65b083d07c3 expref: -99 pid: 12577 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00080000:2.0:1789761039.678450:0:2392:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9018d070c380 x1876699503990272/t0(0) o103->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:17/18 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1789761039.678474:0:6165:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9018c54e9800 lock: ffff9018d1806600/0xeac59f702d4e6ec4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xfa2da65b083d0761 expref: -99 pid: 12570 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789761039.678489:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761039.678491:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761039.678492:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761040.702221:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761041.726237:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761042.750235:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761043.774275:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761044.158175:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761044.158185:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761044.158189:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761044.158201:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 970 00000100:00080000:0.0:1789761044.158213:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761044.158220:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761044.158266:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699503990400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761044.158298:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761044.158302:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761044.158304:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761044.158309:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 970 00000100:00080000:0.0:1789761044.158314:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761044.158328:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761044.158344:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761044.158356:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503990400:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761044.158360:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699503990528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761044.158373:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761044.158375:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761044.158377:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761044.158380:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 970 00000100:00080000:0.0:1789761044.158383:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761044.158386:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761044.158389:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761044.158397:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699503990656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761044.158404:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761044.158407:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761044.158408:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761044.158410:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 970 00000100:00080000:0.0:1789761044.158425:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761044.158428:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761044.158431:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761044.158439:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699503990784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761044.158445:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761044.158456:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761044.158460:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761044.158461:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761044.158465:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 970 00000100:00080000:0.0:1789761044.158467:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761044.158470:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761044.158474:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761044.158482:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699503990912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761044.158489:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761044.158492:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761044.158493:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761044.158496:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 970 00000100:00080000:0.0:1789761044.158498:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761044.158502:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761044.158506:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761044/real 1789761044] req@ffff9018c8b98380 x1876699503990400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789761044.158507:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761044.158519:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699503991040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761044.158534:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699503990400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761044.158548:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761044.158554:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761044.158558:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503990400:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761044.158586:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503990528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761044.158610:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761044.158622:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761044/real 1789761044] req@ffff9018c8b9a300 x1876699503990528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761044.158634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699503990528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761044.158643:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761044.158647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761044.158652:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503990528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761044.158658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503990656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761044.158676:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761044.158686:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761044/real 1789761044] req@ffff9018c8b99500 x1876699503990656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761044.158698:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699503990656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761044.158722:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761044.158726:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761044.158730:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503990656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761044.158736:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503990784:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761044.158758:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761044.158770:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761044/real 1789761044] req@ffff9018c8b99c00 x1876699503990784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761044.158782:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699503990784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761044.158788:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761044.158790:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761044.158792:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503990784:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761044.158797:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503990912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761044.158823:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761044.158831:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761044/real 1789761044] req@ffff9018c8b9ad80 x1876699503990912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761044.158839:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699503990912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761044.158845:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761044.158847:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761044.158849:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503990912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761044.158853:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503991040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761044.158864:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761044.158870:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761044/real 1789761044] req@ffff9018c8b9b100 x1876699503991040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761044.158877:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699503991040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761044.158882:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761044.158884:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761044.158885:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503991040:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761044.798244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761045.822258:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761046.846231:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761047.870159:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761048.894250:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761049.278194:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761049.278201:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761049.278205:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761049.278213:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 975 00000100:00080000:3.0:1789761049.278223:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761049.278229:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761049.278271:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699503991168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761049.278296:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761049.278299:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761049.278301:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761049.278304:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 975 00000100:00080000:3.0:1789761049.278307:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761049.278310:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761049.278319:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761049.278333:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699503991296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761049.278346:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503991168:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761049.278348:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761049.278356:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761049.278359:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761049.278366:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 975 00000100:00080000:3.0:1789761049.278371:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761049.278377:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761049.278383:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761049.278395:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699503991424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761049.278403:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761049.278405:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761049.278406:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761049.278413:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 975 00000800:00000100:1.0:1789761049.278416:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761049.278418:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761049.278423:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761049.278428:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761049.278458:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761049/real 1789761049] req@ffff9018d070dc00 x1876699503991168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761049.278472:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699503991168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761049.278481:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761049.278484:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761049.278487:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503991168:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761049.278508:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699503991552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761049.278518:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761049.278520:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503991296:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761049.278523:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761049.278527:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761049.278531:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 975 00000100:00080000:3.0:1789761049.278534:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761049.278537:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761049.278542:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761049.278548:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761049.278554:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699503991680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761049.278563:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761049/real 1789761049] req@ffff9018d070c000 x1876699503991296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761049.278564:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761049.278572:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761049.278575:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761049.278598:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699503991296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761049.278608:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761049.278612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761049.278616:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503991296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761049.278624:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503991424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761049.278645:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761049.278655:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761049/real 1789761049] req@ffff9018d070fb80 x1876699503991424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761049.278680:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699503991424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761049.278691:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761049.278695:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761049.278699:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503991424:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761049.278707:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 975 00000100:00080000:3.0:1789761049.278711:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761049.278715:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761049.278721:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761049.278724:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503991552:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761049.278736:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699503991808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761049.278751:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761049.278764:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761049/real 1789761049] req@ffff9018d070d880 x1876699503991552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761049.278773:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699503991552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761049.278780:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761049.278782:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761049.278784:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503991552:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761049.278789:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503991680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761049.278804:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761049.278815:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761049/real 1789761049] req@ffff9018d070ce00 x1876699503991680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761049.278824:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699503991680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761049.278830:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761049.278832:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761049.278834:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503991680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761049.278843:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503991808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761049.278853:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761049.278859:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761049/real 1789761049] req@ffff9018d070ca80 x1876699503991808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761049.278867:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699503991808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761049.278879:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761049.278881:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761049.278883:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503991808:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761049.918243:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761050.942199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761051.966202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761052.990194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761054.014209:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761054.398190:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761054.398198:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761054.398202:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761054.398211:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 980 00000100:00080000:0.0:1789761054.398220:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761054.398243:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761054.398277:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699503991936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761054.398302:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761054.398305:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761054.398306:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761054.398309:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 980 00000100:00080000:0.0:1789761054.398311:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761054.398314:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761054.398322:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761054.398350:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699503992064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761054.398360:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761054.398364:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761054.398365:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761054.398368:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 980 00000100:00080000:0.0:1789761054.398371:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761054.398375:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761054.398379:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761054.398390:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699503992192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761054.398398:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503991936:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761054.398437:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761054.398440:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761054.398441:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761054.398444:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 980 00000100:00080000:0.0:1789761054.398446:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761054.398449:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761054.398454:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761054.398462:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699503992320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761054.398469:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761054.398471:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761054.398473:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761054.398476:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 980 00000100:00080000:0.0:1789761054.398504:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761054.398508:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761054.398511:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761054.398518:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699503992448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761054.398524:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761054.398527:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761054.398531:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789761054.398535:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761054.398537:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 980 00000100:00080000:0.0:1789761054.398543:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761054.398547:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761054.398551:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761054.398560:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699503992576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761054.398608:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761054/real 1789761054] req@ffff9018c8b9a680 x1876699503991936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761054.398645:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699503991936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761054.398671:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761054.398676:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761054.398682:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503991936:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761054.398691:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503992064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761054.398721:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761054.398737:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761054/real 1789761054] req@ffff9018c8b98000 x1876699503992064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761054.398752:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699503992064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761054.398769:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761054.398774:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761054.398777:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503992064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761054.398808:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503992192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761054.398832:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761054.398847:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761054/real 1789761054] req@ffff9018c8b9aa00 x1876699503992192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761054.398860:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699503992192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761054.398879:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761054.398882:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761054.398886:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503992192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761054.398893:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503992320:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761054.398914:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761054.398925:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761054/real 1789761054] req@ffff9018c8b9b800 x1876699503992320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761054.398946:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699503992320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761054.398955:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761054.398958:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761054.398963:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503992320:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761054.398970:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503992448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761054.398991:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761054.399004:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761054/real 1789761054] req@ffff9018c8b9b480 x1876699503992448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761054.399025:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699503992448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761054.399067:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761054.399071:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761054.399075:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503992448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761054.399082:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503992576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761054.399102:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761054.399113:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761054/real 1789761054] req@ffff9018c8b99f80 x1876699503992576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761054.399126:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699503992576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761054.399134:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761054.399148:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761054.399152:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503992576:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761055.038187:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761056.062223:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761057.086197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761058.110214:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761059.134200:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761059.518184:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761059.518195:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761059.518200:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761059.518211:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 986 00000100:00080000:0.0:1789761059.518223:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761059.518237:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761059.518280:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699503992704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761059.518306:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761059.518309:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761059.518310:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761059.518313:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 986 00000100:00080000:0.0:1789761059.518316:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761059.518318:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761059.518328:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761059.518337:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699503992832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761059.518350:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761059.518353:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761059.518355:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761059.518357:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 986 00000100:00080000:0.0:1789761059.518359:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761059.518361:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761059.518364:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761059.518372:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699503992960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761059.518380:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761059.518385:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761059.518389:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761059.518390:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503992704:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761059.518393:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 986 00000100:00080000:0.0:1789761059.518396:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761059.518403:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761059.518411:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761059.518419:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699503993088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761059.518426:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761059.518430:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761059.518431:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761059.518434:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 986 00000100:00080000:0.0:1789761059.518437:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761059.518442:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761059.518445:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761059.518455:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699503993216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761059.518460:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761059.518463:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761059.518464:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761059.518468:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 986 00000100:00080000:0.0:1789761059.518471:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761059.518473:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761059.518476:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761059.518484:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761059.518488:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699503993344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761059.518562:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761059/real 1789761059] req@ffff9018c8b99880 x1876699503992704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761059.518597:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699503992704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761059.518611:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761059.518617:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761059.518621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503992704:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761059.518629:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503992832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761059.518654:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761059.518666:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761059/real 1789761059] req@ffff9018c8b98e00 x1876699503992832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761059.518675:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699503992832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761059.518681:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761059.518683:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761059.518685:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503992832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761059.518705:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503992960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761059.518719:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761059.518725:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761059/real 1789761059] req@ffff9018c8b98a80 x1876699503992960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761059.518733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699503992960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761059.518742:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761059.518744:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761059.518746:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503992960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761059.518750:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503993088:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761059.518761:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761059.518767:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761059/real 1789761059] req@ffff9018c8b98700 x1876699503993088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761059.518774:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699503993088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761059.518779:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761059.518781:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761059.518782:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503993088:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761059.518786:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503993216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761059.518796:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761059.518802:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761059/real 1789761059] req@ffff9018c8b9bb80 x1876699503993216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761059.518809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699503993216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761059.518815:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761059.518818:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761059.518822:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503993216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761059.518828:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503993344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761059.518848:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761059.518861:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761059/real 1789761059] req@ffff9018c8b99f80 x1876699503993344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761059.518873:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699503993344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761059.518879:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761059.518881:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761059.518884:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503993344:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761060.158199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761061.182171:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761062.206180:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761063.230254:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761064.254218:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761064.638112:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761064.638120:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761064.638124:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761064.638133:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 991 00000100:00080000:1.0:1789761064.638142:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761064.638148:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761064.638179:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699503993472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761064.638207:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761064.638210:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761064.638211:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761064.638214:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 991 00000100:00080000:1.0:1789761064.638216:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761064.638220:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761064.638228:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761064.638236:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699503993600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761064.638244:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761064.638246:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503993472:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761064.638248:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761064.638252:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761064.638256:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 991 00000100:00080000:1.0:1789761064.638258:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761064.638260:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761064.638263:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761064.638271:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699503993728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761064.638277:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761064.638280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761064.638281:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761064.638284:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 991 00000100:00080000:1.0:1789761064.638287:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761064.638289:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761064.638293:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761064.638300:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699503993856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761064.638308:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761064.638312:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761064.638316:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761064.638317:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761064.638320:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 991 00000100:00080000:1.0:1789761064.638323:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761064.638325:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761064.638328:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761064.638335:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699503993984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761064.638340:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761064.638343:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761064.638344:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761064.638347:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 991 00000100:00080000:1.0:1789761064.638349:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761064.638351:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761064.638355:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761064.638362:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761064/real 1789761064] req@ffff9018c66a4380 x1876699503993472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761064.638363:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699503994112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761064.638408:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699503993472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761064.638420:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761064.638425:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761064.638429:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503993472:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761064.638454:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503993600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761064.638477:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761064.638487:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761064/real 1789761064] req@ffff9018c66a7100 x1876699503993600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761064.638495:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699503993600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761064.638506:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761064.638508:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761064.638510:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503993600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761064.638514:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503993728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761064.638526:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761064.638531:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761064/real 1789761064] req@ffff9018c66a7480 x1876699503993728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761064.638539:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699503993728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761064.638544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761064.638545:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761064.638561:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503993728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761064.638565:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503993856:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761064.638581:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761064.638589:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761064/real 1789761064] req@ffff9018c66a4a80 x1876699503993856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761064.638596:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699503993856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761064.638601:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761064.638603:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761064.638606:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503993856:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761064.638610:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503993984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761064.638620:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761064.638626:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761064/real 1789761064] req@ffff9018c66a6a00 x1876699503993984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761064.638633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699503993984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761064.638638:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761064.638639:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761064.638641:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503993984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761064.638644:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503994112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761064.638657:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761064.638662:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761064/real 1789761064] req@ffff9018c66a5880 x1876699503994112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761064.638670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699503994112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761064.638674:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761064.638676:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761064.638678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503994112:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761065.278195:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761066.302192:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761067.326194:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761068.350177:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761069.374249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761069.758132:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761069.758143:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761069.758147:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761069.758159:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 996 00000100:00080000:2.0:1789761069.758170:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761069.758177:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761069.758219:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699503994240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761069.758244:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761069.758247:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761069.758249:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761069.758253:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 996 00000100:00080000:2.0:1789761069.758256:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761069.758259:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761069.758269:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761069.758284:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699503994368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761069.758285:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503994240:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761069.758300:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761069.758302:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761069.758303:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761069.758306:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 996 00000100:00080000:2.0:1789761069.758308:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761069.758310:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761069.758314:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761069.758324:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699503994496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761069.758332:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761069.758334:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761069.758336:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761069.758338:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 996 00000100:00080000:2.0:1789761069.758341:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761069.758343:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761069.758347:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761069.758355:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699503994624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761069.758362:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761069.758364:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761069.758365:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761069.758368:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 996 00000100:00080000:2.0:1789761069.758371:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761069.758376:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761069.758379:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761069.758383:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761069.758391:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699503994752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761069.758396:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761069.758399:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761069.758401:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761069.758404:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 996 00000100:00080000:2.0:1789761069.758410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761069.758413:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761069.758416:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761069.758423:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699503994880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761069.758448:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761069/real 1789761069] req@ffff9018d0f27480 x1876699503994240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761069.758467:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699503994240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761069.758481:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761069.758485:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761069.758490:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503994240:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761069.758531:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503994368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761069.758554:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761069.758567:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761069/real 1789761069] req@ffff9018d0f24700 x1876699503994368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761069.758581:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699503994368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761069.758590:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761069.758594:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761069.758598:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503994368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761069.758605:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503994496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761069.758625:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761069.758635:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761069/real 1789761069] req@ffff9018d0f26680 x1876699503994496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761069.758648:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699503994496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761069.758657:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761069.758661:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761069.758665:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503994496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761069.758672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503994624:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761069.758694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761069.758703:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761069/real 1789761069] req@ffff9018d0f24e00 x1876699503994624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761069.758712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699503994624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761069.758718:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761069.758720:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761069.758722:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503994624:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761069.758727:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503994752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761069.758740:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761069.758746:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761069/real 1789761069] req@ffff9018d0f24000 x1876699503994752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761069.758753:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699503994752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761069.758759:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761069.758760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761069.758762:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503994752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761069.758766:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503994880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761069.758777:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761069.758783:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761069/real 1789761069] req@ffff9018d0f25500 x1876699503994880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761069.758791:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699503994880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761069.758797:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761069.758798:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761069.758800:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503994880:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761070.398204:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761071.422144:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761072.446156:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761073.470161:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761074.494158:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761074.878110:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761074.878121:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761074.878133:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761074.878143:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1001 00000100:00080000:2.0:1789761074.878156:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761074.878163:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761074.878205:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699503995008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761074.878231:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761074.878234:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761074.878235:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761074.878238:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1001 00000100:00080000:2.0:1789761074.878241:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761074.878243:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761074.878252:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761074.878263:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699503995136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761074.878275:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761074.878279:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761074.878280:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761074.878283:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1001 00000100:00080000:2.0:1789761074.878286:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761074.878289:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761074.878292:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761074.878301:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699503995264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761074.878309:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761074.878314:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761074.878317:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761074.878320:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503995008:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761074.878323:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1001 00000100:00080000:2.0:1789761074.878333:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761074.878338:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761074.878342:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761074.878352:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699503995392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761074.878364:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761074.878368:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761074.878369:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761074.878374:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1001 00000100:00080000:2.0:1789761074.878379:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761074.878385:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761074.878392:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761074.878396:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761074.878421:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699503995520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761074.878436:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761074.878440:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761074.878442:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761074.878448:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1001 00000100:00100000:1.0:1789761074.878455:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761074/real 1789761074] req@ffff9018d0f26d80 x1876699503995008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761074.878455:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761074.878487:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761074.878495:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761074.878496:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699503995008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761074.878513:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789761074.878514:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699503995648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761074.878521:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761074.878534:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503995008:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761074.878552:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503995136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761074.878578:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761074.878588:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761074/real 1789761074] req@ffff9018d0f27800 x1876699503995136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761074.878597:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699503995136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761074.878607:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761074.878610:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761074.878612:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503995136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761074.878634:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503995264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761074.878645:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761074.878651:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761074/real 1789761074] req@ffff9018d0f25180 x1876699503995264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761074.878658:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699503995264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761074.878664:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761074.878666:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761074.878667:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503995264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761074.878671:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503995392:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761074.878682:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761074.878688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761074/real 1789761074] req@ffff9018d0f24a80 x1876699503995392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761074.878696:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699503995392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761074.878702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761074.878705:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761074.878709:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503995392:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761074.878716:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503995520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761074.878735:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761074.878747:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761074/real 1789761074] req@ffff9018d0f24380 x1876699503995520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761074.878759:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699503995520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761074.878768:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761074.878770:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761074.878772:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503995520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761074.878777:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503995648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761074.878792:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761074.878799:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761074/real 1789761074] req@ffff9018d0f26300 x1876699503995648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761074.878807:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699503995648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761074.878812:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761074.878814:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761074.878816:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503995648:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761075.518202:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761076.542206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761077.566237:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761078.590224:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761079.614275:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761079.998090:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761079.998102:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761079.998107:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761079.998118:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1006 00000100:00080000:2.0:1789761079.998129:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761079.998136:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761079.998181:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699503995776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761079.998221:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761079.998225:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761079.998227:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761079.998232:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1006 00000100:00080000:2.0:1789761079.998236:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761079.998241:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761079.998250:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761079.998263:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699503995904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761079.998272:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761079.998274:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761079.998276:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761079.998279:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1006 00000100:00080000:2.0:1789761079.998281:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761079.998284:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761079.998287:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761079.998296:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699503996032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761079.998303:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503995776:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761079.998306:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761079.998319:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761079.998331:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761079.998336:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1006 00000100:00080000:2.0:1789761079.998340:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761079.998344:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761079.998349:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761079.998374:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699503996160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761079.998382:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761079.998385:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761079.998386:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761079.998390:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1006 00000100:00080000:2.0:1789761079.998395:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761079.998399:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761079.998402:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761079.998409:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761079.998420:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699503996288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761079.998429:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761079.998442:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761079.998444:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761079.998447:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1006 00000100:00080000:2.0:1789761079.998451:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761079.998454:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761079.998458:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761079.998484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699503996416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761079.998499:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761079/real 1789761079] req@ffff9018d0f27100 x1876699503995776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761079.998528:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699503995776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761079.998541:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761079.998546:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761079.998550:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503995776:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761079.998559:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503995904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761079.998583:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761079.998593:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761079/real 1789761079] req@ffff9018d0f26a00 x1876699503995904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761079.998602:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699503995904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761079.998617:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761079.998619:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761079.998621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503995904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761079.998640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503996032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761079.998652:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761079.998658:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761079/real 1789761079] req@ffff9018d0f25c00 x1876699503996032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761079.998666:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699503996032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761079.998676:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761079.998678:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761079.998680:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503996032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761079.998684:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503996160:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761079.998705:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761079.998710:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761079/real 1789761079] req@ffff9018d0f25f80 x1876699503996160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761079.998717:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699503996160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761079.998722:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761079.998724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761079.998726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503996160:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761079.998729:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503996288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761079.998739:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761079.998748:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761079/real 1789761079] req@ffff9018d0f26300 x1876699503996288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761079.998770:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699503996288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761079.998778:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761079.998782:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761079.998785:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503996288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761079.998792:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503996416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761079.998812:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761079.998821:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761079/real 1789761079] req@ffff9018d0f24380 x1876699503996416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761079.998829:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699503996416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761079.998834:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761079.998836:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761079.998838:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503996416:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761080.638283:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761081.662196:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761082.686220:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761083.710242:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761084.734200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761085.118130:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761085.118137:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761085.118141:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761085.118149:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1011 00000100:00080000:2.0:1789761085.118158:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761085.118163:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761085.118196:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699503996544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761085.118222:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761085.118225:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761085.118226:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761085.118229:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1011 00000100:00080000:2.0:1789761085.118231:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761085.118234:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761085.118241:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761085.118249:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699503996672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761085.118256:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761085.118258:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761085.118260:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761085.118262:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1011 00000100:00080000:2.0:1789761085.118265:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761085.118267:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761085.118270:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761085.118277:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699503996800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761085.118284:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761085.118287:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761085.118288:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761085.118291:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1011 00000100:00080000:2.0:1789761085.118294:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761085.118296:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503996544:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761085.118300:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761085.118306:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761085.118314:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699503996928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761085.118321:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761085.118323:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761085.118325:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761085.118327:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1011 00000100:00080000:2.0:1789761085.118330:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761085.118332:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761085.118335:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761085.118343:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699503997056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761085.118354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789761085.118358:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761085.118360:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761085.118363:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761085.118366:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1011 00000100:00080000:2.0:1789761085.118370:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761085.118373:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761085.118376:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761085.118385:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699503997184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761085.118421:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761085/real 1789761085] req@ffff9018d0f24a80 x1876699503996544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761085.118438:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699503996544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761085.118449:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761085.118453:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761085.118458:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503996544:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761085.118466:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503996672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761085.118490:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761085.118503:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761085/real 1789761085] req@ffff9018d0f25180 x1876699503996672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761085.118512:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699503996672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761085.118518:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761085.118521:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761085.118523:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503996672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761085.118544:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503996800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761085.118570:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761085.118580:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761085/real 1789761085] req@ffff9018d0f27800 x1876699503996800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761085.118591:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699503996800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761085.118597:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761085.118600:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761085.118602:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503996800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761085.118606:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503996928:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761085.118619:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761085.118631:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761085/real 1789761085] req@ffff9018d0f26d80 x1876699503996928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761085.118639:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699503996928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761085.118645:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761085.118647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761085.118648:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503996928:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761085.118652:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503997056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761085.118663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761085.118671:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761085/real 1789761085] req@ffff9018d0f25500 x1876699503997056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761085.118679:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699503997056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761085.118685:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761085.118687:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761085.118689:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503997056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761085.118694:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503997184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761085.118707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761085.118715:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761085/real 1789761085] req@ffff9018d0f24000 x1876699503997184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761085.118728:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699503997184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761085.118735:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761085.118738:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761085.118741:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503997184:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761085.758192:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761086.782203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761087.806152:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761088.830216:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761089.854205:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761090.238167:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761090.238174:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761090.238178:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761090.238186:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1016 00000100:00080000:0.0:1789761090.238196:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761090.238202:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761090.238232:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699503997312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761090.238256:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761090.238259:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761090.238260:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761090.238263:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1016 00000100:00080000:0.0:1789761090.238265:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761090.238267:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761090.238275:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761090.238285:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699503997440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761090.238292:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761090.238294:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761090.238296:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761090.238299:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1016 00000100:00080000:0.0:1789761090.238300:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761090.238303:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761090.238306:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761090.238316:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503997312:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761090.238324:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699503997568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761090.238336:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761090.238339:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761090.238340:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761090.238343:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1016 00000100:00080000:0.0:1789761090.238345:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761090.238347:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761090.238351:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761090.238375:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699503997696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761090.238386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761090.238389:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761090.238391:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789761090.238399:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761090.238404:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1016 00000100:00080000:0.0:1789761090.238411:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761090.238416:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761090.238422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761090.238443:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761090/real 1789761090] req@ffff9018c8b9b480 x1876699503997312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761090.238452:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699503997824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761090.238462:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699503997312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761090.238474:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761090.238478:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761090.238479:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761090.238485:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761090.238489:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761090.238494:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503997312:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761090.238496:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1016 00000100:00080000:0.0:1789761090.238505:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761090.238510:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503997440:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761090.238511:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761090.238518:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761090.238531:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699503997952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761090.238547:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761090.238563:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761090/real 1789761090] req@ffff9018c8b9b800 x1876699503997440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761090.238576:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699503997440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761090.238585:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761090.238588:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761090.238592:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503997440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761090.238615:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503997568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761090.238631:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761090.238642:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761090/real 1789761090] req@ffff9018c8b9aa00 x1876699503997568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761090.238654:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699503997568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761090.238663:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761090.238666:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761090.238670:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503997568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761090.238676:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503997696:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761090.238698:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761090.238709:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761090/real 1789761090] req@ffff9018c8b98000 x1876699503997696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761090.238721:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699503997696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761090.238730:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761090.238733:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761090.238737:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503997696:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761090.238742:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503997824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761090.238758:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761090.238766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761090/real 1789761090] req@ffff9018c8b9a680 x1876699503997824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761090.238777:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699503997824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761090.238786:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761090.238790:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761090.238793:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503997824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761090.238798:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503997952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761090.238813:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761090.238821:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761090/real 1789761090] req@ffff9018c8b9b100 x1876699503997952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761090.238833:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699503997952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761090.238841:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761090.238844:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761090.238847:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503997952:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761090.878254:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761091.902211:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761092.926240:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761093.950152:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761094.974181:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761095.358093:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761095.358100:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761095.358103:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761095.358111:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1021 00000100:00080000:0.0:1789761095.358119:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761095.358125:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761095.358153:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699503998080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761095.358177:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761095.358181:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761095.358183:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761095.358187:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1021 00000100:00080000:0.0:1789761095.358191:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761095.358196:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761095.358206:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761095.358220:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503998080:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761095.358222:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699503998208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761095.358237:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761095.358240:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761095.358241:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761095.358244:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1021 00000100:00080000:0.0:1789761095.358248:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761095.358251:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761095.358255:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761095.358265:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699503998336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761095.358272:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761095.358275:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761095.358276:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761095.358278:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1021 00000100:00080000:0.0:1789761095.358281:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761095.358283:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761095.358292:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761095.358306:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699503998464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761095.358316:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503998208:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761095.358332:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761095.358338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761095.358340:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761095.358346:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1021 00000100:00080000:0.0:1789761095.358351:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761095.358357:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761095.358364:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761095.358365:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503998336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761095.358380:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699503998592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761095.358388:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761095.358391:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761095.358393:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761095.358397:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503998464:192.168.203.158@tcp:8:kworker.0 00000100:00080000:0.0:1789761095.358398:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1021 00000100:00080000:0.0:1789761095.358405:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761095.358409:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761095.358412:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761095.358423:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699503998720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761095.358425:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503998592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761095.358455:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503998720:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761095.998162:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761097.022224:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761098.046263:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761098.430361:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:3.0:1789761098.430409:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761098.430438:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761098.430474:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761098.430483:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761098.430491:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761098.430500:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761098.430512:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789761098.430514:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761095/real 1789761098] req@ffff9018c8b9a300 x1876699503998464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761098.430553:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699503998464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761098.430579:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761098.430585:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761098.430590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503998464:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761098.430599:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761095/real 1789761098] req@ffff9018c8b98380 x1876699503998592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761098.430613:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699503998592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761098.430622:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761098.430626:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761098.430629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503998592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761098.430636:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761095/real 1789761098] req@ffff9018c8b99c00 x1876699503998208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761098.430648:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699503998208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761098.430656:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761098.430660:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761098.430673:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503998208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761098.430681:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761095/real 1789761098] req@ffff9018c8b99500 x1876699503998336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761098.430693:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699503998336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761098.430702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761098.430706:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761098.430710:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503998336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761098.430719:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761095/real 1789761098] req@ffff9018c8b9ad80 x1876699503998080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761098.430733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699503998080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761098.430742:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761098.430746:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761098.430749:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503998080:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761098.430789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761095/real 1789761098] req@ffff9018c8b9b100 x1876699503998720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761098.430804:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699503998720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761098.430813:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761098.430817:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761098.430821:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503998720:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761099.070233:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761100.094224:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761100.478209:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761100.478217:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761100.478221:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761100.478229:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1027 00000100:00080000:1.0:1789761100.478237:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761100.478243:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761100.478272:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699503998848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761100.478311:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761100.478314:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761100.478316:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761100.478319:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1027 00000100:00080000:1.0:1789761100.478322:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761100.478326:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761100.478334:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761100.478345:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699503998976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761100.478356:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761100.478359:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761100.478361:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761100.478367:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1027 00000100:00080000:1.0:1789761100.478370:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761100.478375:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761100.478375:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503998848:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789761100.478386:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761100.478401:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699503999104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761100.478413:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761100.478416:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761100.478418:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761100.478421:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1027 00000100:00080000:1.0:1789761100.478424:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761100.478428:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761100.478434:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789761100.478439:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761100.478450:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699503999232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761100.478462:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761100.478466:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761100.478468:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761100.478473:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1027 00000100:00080000:1.0:1789761100.478478:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761100.478482:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761100/real 1789761100] req@ffff9018c66a5880 x1876699503998848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761100.478486:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761100.478495:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761100.478511:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699503998848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761100.478513:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699503999360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761100.478548:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761100.478552:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761100.478553:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761100.478558:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761100.478562:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789761100.478567:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1027 00000100:00080000:1.0:1789761100.478576:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761100.478582:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761100.478587:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503998848:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789761100.478590:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761100.478608:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699503999488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761100.478627:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503998976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761100.478657:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761100.478674:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761100/real 1789761100] req@ffff9018c66a6a00 x1876699503998976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761100.478688:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699503998976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761100.478697:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761100.478701:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761100.478705:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503998976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761100.478711:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503999104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761100.478733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761100.478745:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761100/real 1789761100] req@ffff9018c66a4a80 x1876699503999104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761100.478757:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699503999104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761100.478765:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761100.478768:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761100.478771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503999104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761100.478778:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503999232:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761100.478800:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761100.478812:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761100/real 1789761100] req@ffff9018c66a7480 x1876699503999232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761100.478825:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699503999232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761100.478834:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761100.478837:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761100.478840:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503999232:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761100.478847:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503999360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761100.478870:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761100.478882:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761100/real 1789761100] req@ffff9018c66a7100 x1876699503999360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761100.478894:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699503999360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761100.478903:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761100.478906:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761100.478909:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503999360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761100.478915:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503999488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761100.478941:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761100.478960:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761100/real 1789761100] req@ffff9018c66a4380 x1876699503999488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761100.478972:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699503999488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761100.478984:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761100.478987:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761100.478991:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699503999488:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761101.118269:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761102.142330:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761103.166218:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761104.190177:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761105.214281:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761105.598200:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761105.598215:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761105.598220:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761105.598230:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1032 00000100:00080000:2.0:1789761105.598257:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761105.598264:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761105.598311:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699503999616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761105.598339:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761105.598342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761105.598343:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761105.598347:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1032 00000100:00080000:2.0:1789761105.598350:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761105.598353:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761105.598363:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761105.598373:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699503999744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761105.598379:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503999616:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761105.598382:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761105.598388:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761105.598389:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761105.598392:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1032 00000100:00080000:2.0:1789761105.598394:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761105.598396:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761105.598399:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761105.598411:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699503999872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761105.598421:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761105.598424:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761105.598426:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761105.598431:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1032 00000100:00080000:2.0:1789761105.598435:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761105.598440:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761105.598447:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789761105.598458:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761105.598464:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504000000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761105.598476:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761105.598480:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761105.598483:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761105.598488:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1032 00000100:00080000:2.0:1789761105.598492:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761105.598497:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761105.598504:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761105.598518:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504000128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761105.598530:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761105/real 1789761105] req@ffff9018d0f24e00 x1876699503999616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761105.598533:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761105.598542:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761105.598547:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761105.598555:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1032 00000100:00080000:2.0:1789761105.598560:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761105.598565:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761105.598571:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761105.598576:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699503999616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761105.598586:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504000256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761105.598595:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761105.598602:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761105.598608:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699503999616:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761105.598637:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503999744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761105.598664:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761105.598680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761105/real 1789761105] req@ffff9018d0f26680 x1876699503999744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761105.598694:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699503999744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761105.598710:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761105.598714:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761105.598718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503999744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761105.598725:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503999872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761105.598751:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761105.598765:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761105/real 1789761105] req@ffff9018d0f24700 x1876699503999872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761105.598779:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699503999872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761105.598790:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761105.598794:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761105.598796:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699503999872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761105.598801:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504000000:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761105.598819:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761105.598827:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761105/real 1789761105] req@ffff9018d0f27480 x1876699504000000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761105.598835:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504000000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761105.598841:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761105.598843:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761105.598846:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504000000:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761105.598850:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504000128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761105.598862:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761105.598868:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761105/real 1789761105] req@ffff9018d0f24000 x1876699504000128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761105.598875:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504000128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761105.598880:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761105.598883:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761105.598884:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504000128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761105.598888:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504000256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761105.598898:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761105.598904:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761105/real 1789761105] req@ffff9018d0f25500 x1876699504000256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761105.598912:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504000256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761105.598916:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761105.598918:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761105.598920:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504000256:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761106.238249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761107.262189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761108.286190:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761109.310138:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761110.334183:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761110.718096:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761110.718104:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761110.718108:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761110.718116:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1037 00000100:00080000:2.0:1789761110.718124:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761110.718130:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761110.718161:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504000384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761110.718183:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761110.718186:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761110.718188:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761110.718191:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1037 00000100:00080000:2.0:1789761110.718193:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761110.718196:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761110.718216:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761110.718231:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504000512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761110.718241:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761110.718245:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761110.718246:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761110.718249:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1037 00000100:00080000:2.0:1789761110.718252:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761110.718255:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761110.718260:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761110.718267:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504000640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761110.718278:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504000384:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761110.718280:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761110.718296:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761110.718298:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761110.718302:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1037 00000100:00080000:2.0:1789761110.718304:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761110.718306:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761110.718309:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761110.718318:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504000768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761110.718325:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761110.718328:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761110.718329:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761110.718332:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1037 00000100:00080000:2.0:1789761110.718334:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761110.718337:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761110.718341:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761110.718349:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504000896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761110.718355:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761110.718358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761110.718359:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761110.718362:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1037 00000100:00080000:2.0:1789761110.718364:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761110.718367:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761110.718372:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761110.718380:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761110.718385:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504001024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761110.718439:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761110/real 1789761110] req@ffff9018d0f26d80 x1876699504000384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761110.718457:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504000384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761110.718468:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761110.718474:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761110.718479:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504000384:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761110.718490:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504000512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761110.718517:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761110.718533:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761110/real 1789761110] req@ffff9018d0f27800 x1876699504000512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761110.718544:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504000512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761110.718551:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761110.718553:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761110.718556:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504000512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761110.718582:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504000640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761110.718598:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761110.718606:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761110/real 1789761110] req@ffff9018d0f25180 x1876699504000640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761110.718614:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504000640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761110.718624:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761110.718627:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761110.718629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504000640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761110.718633:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504000768:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761110.718644:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761110.718651:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761110/real 1789761110] req@ffff9018d0f24a80 x1876699504000768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761110.718658:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504000768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761110.718664:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761110.718666:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761110.718668:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504000768:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761110.718672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504000896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761110.718683:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761110.718689:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761110/real 1789761110] req@ffff9018d0f24380 x1876699504000896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761110.718696:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504000896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761110.718701:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761110.718704:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761110.718708:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504000896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761110.718714:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504001024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761110.718734:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761110.718746:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761110/real 1789761110] req@ffff9018d0f26300 x1876699504001024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761110.718759:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504001024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761110.718768:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761110.718772:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761110.718775:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504001024:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761111.358208:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761112.382143:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761113.406159:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761114.430243:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761115.454184:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761115.838209:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761115.838234:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761115.838240:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761115.838253:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1042 00000100:00080000:0.0:1789761115.838268:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761115.838278:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761115.838343:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504001152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761115.838378:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761115.838383:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761115.838385:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761115.838390:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1042 00000100:00080000:0.0:1789761115.838393:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761115.838398:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761115.838408:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761115.838432:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504001280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761115.838440:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761115.838443:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761115.838445:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761115.838451:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1042 00000100:00100000:1.0:1789761115.838453:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504001152:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761115.838454:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761115.838461:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761115.838466:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761115.838475:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504001408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761115.838482:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761115.838484:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761115.838486:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761115.838488:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1042 00000100:00080000:0.0:1789761115.838491:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761115.838494:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761115.838498:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761115.838506:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504001536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761115.838514:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761115.838517:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761115.838518:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761115.838521:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1042 00000100:00080000:0.0:1789761115.838523:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761115.838526:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761115.838529:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761115.838534:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761115.838538:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504001664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761115.838552:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761115.838555:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761115.838556:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761115.838559:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1042 00000100:00080000:0.0:1789761115.838562:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761115.838564:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761115.838568:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761115.838575:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504001792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761115.838611:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761115/real 1789761115] req@ffff9018c8b9a680 x1876699504001152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761115.838629:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504001152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761115.838638:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761115.838641:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761115.838645:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504001152:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761115.838668:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504001280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761115.838684:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761115.838694:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761115/real 1789761115] req@ffff9018c8b98000 x1876699504001280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761115.838702:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504001280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761115.838708:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761115.838710:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761115.838713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504001280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761115.838716:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504001408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761115.838735:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761115.838747:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761115/real 1789761115] req@ffff9018c8b9aa00 x1876699504001408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761115.838760:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504001408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761115.838769:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761115.838773:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761115.838776:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504001408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761115.838781:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504001536:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761115.838799:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761115.838806:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761115/real 1789761115] req@ffff9018c8b9b800 x1876699504001536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761115.838814:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504001536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761115.838821:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761115.838823:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761115.838825:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504001536:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761115.838828:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504001664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761115.838839:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761115.838845:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761115/real 1789761115] req@ffff9018c8b9b480 x1876699504001664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761115.838852:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504001664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761115.838857:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761115.838859:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761115.838861:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504001664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761115.838864:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504001792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761115.838874:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761115.838880:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761115/real 1789761115] req@ffff9018c8b99f80 x1876699504001792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761115.838888:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504001792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761115.838893:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761115.838895:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761115.838898:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504001792:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761116.478255:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761117.502240:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761118.526263:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761119.550231:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761120.574248:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761120.958162:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761120.958185:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761120.958191:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761120.958205:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1047 00000100:00080000:0.0:1789761120.958219:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761120.958228:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761120.958268:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504001920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761120.958292:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761120.958295:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761120.958297:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761120.958301:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1047 00000100:00080000:0.0:1789761120.958305:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761120.958308:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761120.958318:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761120.958329:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504001920:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761120.958330:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504002048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761120.958346:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761120.958349:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761120.958350:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761120.958353:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1047 00000100:00080000:0.0:1789761120.958356:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761120.958359:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761120.958365:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761120.958376:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504002176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761120.958383:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761120.958385:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761120.958387:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761120.958389:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1047 00000100:00080000:0.0:1789761120.958398:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761120.958402:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761120.958405:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761120.958413:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504002304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761120.958419:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761120.958425:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761120.958428:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761120.958430:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761120.958433:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1047 00000100:00080000:0.0:1789761120.958435:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761120.958438:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761120.958441:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761120.958448:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504002432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761120.958455:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761120.958457:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761120.958459:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761120.958463:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1047 00000100:00080000:0.0:1789761120.958465:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761120.958468:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761120.958471:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761120.958476:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761120/real 1789761120] req@ffff9018c8b9bb80 x1876699504001920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761120.958482:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504002560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761120.958504:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504001920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761120.958516:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761120.958520:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761120.958525:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504001920:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761120.958549:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504002048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761120.958568:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761120.958577:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761120/real 1789761120] req@ffff9018c8b98700 x1876699504002048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761120.958585:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504002048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761120.958591:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761120.958593:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761120.958595:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504002048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761120.958599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504002176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761120.958609:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761120.958616:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761120/real 1789761120] req@ffff9018c8b98a80 x1876699504002176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761120.958624:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504002176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761120.958629:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761120.958630:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761120.958632:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504002176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761120.958635:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504002304:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761120.958647:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761120.958652:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761120/real 1789761120] req@ffff9018c8b98e00 x1876699504002304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761120.958660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504002304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761120.958665:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761120.958667:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761120.958669:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504002304:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761120.958672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504002432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761120.958681:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761120.958687:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761120/real 1789761120] req@ffff9018c8b99880 x1876699504002432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761120.958702:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504002432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761120.958709:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761120.958711:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761120.958713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504002432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761120.958717:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504002560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761120.958729:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761120.958735:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761120/real 1789761120] req@ffff9018c8b99f80 x1876699504002560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761120.958743:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504002560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761120.958748:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761120.958750:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761120.958752:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504002560:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761121.598216:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761122.622208:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761123.646268:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761124.670233:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761125.694233:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761126.078166:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761126.078173:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761126.078177:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761126.078186:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1052 00000100:00080000:2.0:1789761126.078195:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761126.078202:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761126.078234:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504002688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761126.078257:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761126.078260:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761126.078262:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761126.078267:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1052 00000100:00080000:2.0:1789761126.078270:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761126.078274:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761126.078284:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761126.078297:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504002816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761126.078307:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761126.078310:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761126.078312:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761126.078315:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504002688:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761126.078315:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1052 00000100:00080000:2.0:1789761126.078319:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761126.078322:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761126.078327:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761126.078336:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504002944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761126.078344:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761126.078347:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761126.078348:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761126.078352:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1052 00000100:00080000:2.0:1789761126.078354:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761126.078357:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761126.078361:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761126.078369:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504003072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761126.078377:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761126.078378:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761126.078381:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761126.078382:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761126.078385:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1052 00000100:00080000:2.0:1789761126.078388:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761126.078391:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761126.078396:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761126.078405:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504003200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761126.078413:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761126/real 1789761126] req@ffff9018d0f25f80 x1876699504002688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761126.078414:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761126.078417:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761126.078419:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761126.078422:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1052 00000100:00080000:2.0:1789761126.078426:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761126.078428:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761126.078430:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504002688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761126.078433:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761126.078441:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789761126.078441:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504003328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761126.078446:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761126.078450:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504002688:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761126.078473:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504002816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761126.078495:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761126.078505:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761126/real 1789761126] req@ffff9018d0f25c00 x1876699504002816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761126.078519:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504002816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761126.078527:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761126.078530:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761126.078534:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504002816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761126.078540:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504002944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761126.078560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761126.078571:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761126/real 1789761126] req@ffff9018d0f26a00 x1876699504002944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761126.078580:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504002944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761126.078585:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761126.078588:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761126.078590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504002944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761126.078593:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504003072:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761126.078606:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761126.078613:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761126/real 1789761126] req@ffff9018d0f27100 x1876699504003072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761126.078620:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504003072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761126.078626:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761126.078627:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761126.078629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504003072:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761126.078633:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504003200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761126.078642:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761126.078648:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761126/real 1789761126] req@ffff9018d0f26300 x1876699504003200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761126.078655:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504003200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761126.078660:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761126.078662:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761126.078664:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504003200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761126.078667:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504003328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761126.078675:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761126.078680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761126/real 1789761126] req@ffff9018d0f24380 x1876699504003328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761126.078687:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504003328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761126.078701:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761126.078703:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761126.078705:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504003328:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761126.718212:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761127.742295:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761128.766203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761129.790199:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761130.814167:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761131.198111:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761131.198119:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761131.198122:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761131.198130:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1057 00000100:00080000:2.0:1789761131.198139:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761131.198145:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761131.198175:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504003456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761131.198196:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761131.198199:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761131.198200:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761131.198203:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1057 00000100:00080000:2.0:1789761131.198205:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761131.198207:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761131.198215:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761131.198224:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504003584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761131.198232:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761131.198234:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761131.198235:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761131.198238:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1057 00000100:00080000:2.0:1789761131.198240:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761131.198245:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504003456:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761131.198245:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761131.198253:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761131.198259:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504003712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761131.198267:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761131.198269:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761131.198270:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761131.198273:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1057 00000100:00080000:2.0:1789761131.198275:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761131.198277:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761131.198281:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761131.198288:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504003840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761131.198295:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761131.198297:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761131.198299:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761131.198301:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1057 00000100:00080000:2.0:1789761131.198304:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761131.198307:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789761131.198314:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789761131.198314:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761131.198324:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504003968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761131.198332:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761131.198334:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761131.198336:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761131.198338:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1057 00000100:00080000:2.0:1789761131.198340:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761131.198342:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761131.198345:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761131.198352:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504004096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761131.198371:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761131/real 1789761131] req@ffff9018d0f24a80 x1876699504003456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761131.198392:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504003456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761131.198404:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761131.198407:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761131.198410:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504003456:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761131.198418:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504003584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761131.198438:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761131.198450:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761131/real 1789761131] req@ffff9018d0f25180 x1876699504003584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761131.198463:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504003584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761131.198472:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761131.198475:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761131.198479:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504003584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761131.198503:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504003712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761131.198525:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761131.198537:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761131/real 1789761131] req@ffff9018d0f27800 x1876699504003712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761131.198549:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504003712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761131.198558:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761131.198561:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761131.198565:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504003712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761131.198571:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504003840:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761131.198588:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761131.198599:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761131/real 1789761131] req@ffff9018d0f26d80 x1876699504003840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761131.198612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504003840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761131.198620:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761131.198623:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761131.198627:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504003840:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761131.198633:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504003968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761131.198654:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761131.198664:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761131/real 1789761131] req@ffff9018d0f25500 x1876699504003968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761131.198677:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504003968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761131.198685:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761131.198688:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761131.198691:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504003968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761131.198697:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504004096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761131.198712:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761131.198720:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761131/real 1789761131] req@ffff9018d0f24000 x1876699504004096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761131.198731:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504004096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761131.198739:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761131.198742:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761131.198745:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504004096:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761131.838146:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761132.862146:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761133.886187:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761134.910171:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761135.934185:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761136.318110:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761136.318120:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761136.318125:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761136.318135:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1062 00000100:00080000:2.0:1789761136.318145:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761136.318152:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761136.318185:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504004224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761136.318214:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761136.318218:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761136.318220:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761136.318225:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1062 00000100:00080000:2.0:1789761136.318229:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761136.318233:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761136.318242:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504004224:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789761136.318242:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761136.318259:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504004352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761136.318271:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761136.318275:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761136.318277:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761136.318281:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1062 00000100:00080000:2.0:1789761136.318285:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761136.318289:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761136.318293:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761136.318299:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761136.318313:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504004480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761136.318324:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761136.318328:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761136.318330:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761136.318333:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761136/real 1789761136] req@ffff9018d0f27480 x1876699504004224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761136.318337:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1062 00000100:00080000:2.0:1789761136.318345:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761136.318351:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761136.318356:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504004224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761136.318356:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761136.318371:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504004608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761136.318382:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761136.318384:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761136.318391:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789761136.318391:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761136.318395:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761136.318398:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504004224:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761136.318403:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1062 00000100:00080000:2.0:1789761136.318408:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761136.318412:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761136.318416:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761136.318422:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504004352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761136.318428:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504004736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761136.318440:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789761136.318446:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761136.318447:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761136.318452:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761136.318457:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1062 00000100:00080000:2.0:1789761136.318461:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761136.318462:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761136/real 1789761136] req@ffff9018d0f24700 x1876699504004352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761136.318467:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761136.318476:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761136.318482:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504004352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761136.318490:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504004864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761136.318492:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761136.318500:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761136.318507:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504004352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761136.318519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504004480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761136.318539:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761136.318548:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761136/real 1789761136] req@ffff9018d0f26680 x1876699504004480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761136.318555:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504004480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761136.318560:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761136.318562:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761136.318564:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504004480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761136.318568:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504004608:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761136.318579:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761136.318585:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761136/real 1789761136] req@ffff9018d0f24e00 x1876699504004608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761136.318592:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504004608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761136.318597:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761136.318599:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761136.318601:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504004608:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761136.318611:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504004736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761136.318620:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761136.318626:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761136/real 1789761136] req@ffff9018d0f27480 x1876699504004736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761136.318634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504004736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761136.318639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761136.318640:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761136.318642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504004736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761136.318645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504004864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761136.318654:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761136.318660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761136/real 1789761136] req@ffff9018d0f24000 x1876699504004864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761136.318668:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504004864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761136.318673:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761136.318674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761136.318676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504004864:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761136.958174:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761137.982166:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761139.006162:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761140.030229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761141.054221:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761141.438145:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761141.438156:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761141.438159:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761141.438170:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1067 00000100:00080000:0.0:1789761141.438181:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761141.438188:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761141.438229:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504004992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761141.438258:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761141.438261:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761141.438262:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761141.438266:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1067 00000100:00080000:0.0:1789761141.438270:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761141.438275:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761141.438287:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761141.438304:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504004992:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761141.438306:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504005120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761141.438330:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761141.438334:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761141.438336:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761141.438342:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1067 00000100:00080000:0.0:1789761141.438362:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761141.438367:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761141.438374:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761141.438378:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761141.438394:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504005248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761141.438406:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761141.438410:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761141.438412:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761141.438417:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1067 00000100:00080000:0.0:1789761141.438422:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761141.438427:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761141.438433:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761141.438448:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504005376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761141.438449:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761141/real 1789761141] req@ffff9018c8b9b480 x1876699504004992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761141.438462:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761141.438470:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761141.438473:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761141.438478:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1067 00000100:00080000:0.0:1789761141.438483:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761141.438486:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761141.438490:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761141.438493:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504004992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761141.438500:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504005504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761141.438522:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761141.438524:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761141.438531:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761141.438533:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761141.438535:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761141.438542:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504004992:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761141.438544:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1067 00000100:00080000:0.0:1789761141.438551:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761141.438556:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761141.438562:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761141.438577:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504005120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761141.438578:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504005632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761141.438614:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761141.438629:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761141/real 1789761141] req@ffff9018c8b9b800 x1876699504005120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761141.438642:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504005120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761141.438658:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761141.438663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761141.438667:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504005120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761141.438674:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504005248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761141.438697:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761141.438707:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761141/real 1789761141] req@ffff9018c8b9aa00 x1876699504005248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761141.438715:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504005248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761141.438721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761141.438724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761141.438726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504005248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761141.438730:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504005376:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761141.438743:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761141.438750:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761141/real 1789761141] req@ffff9018c8b98000 x1876699504005376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761141.438758:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504005376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761141.438763:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761141.438765:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761141.438767:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504005376:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761141.438771:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504005504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761141.438783:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761141.438789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761141/real 1789761141] req@ffff9018c8b9a680 x1876699504005504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761141.438796:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504005504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761141.438802:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761141.438803:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761141.438805:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504005504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761141.438818:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504005632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761141.438829:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761141.438835:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761141/real 1789761141] req@ffff9018c8b9b100 x1876699504005632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761141.438843:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504005632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761141.438848:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761141.438850:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761141.438852:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504005632:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761142.078167:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761143.102271:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761144.126208:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761145.150237:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761146.174220:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761146.558148:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761146.558159:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761146.558163:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761146.558173:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1073 00000100:00080000:2.0:1789761146.558184:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761146.558191:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761146.558229:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504005760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761146.558256:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761146.558259:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761146.558260:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761146.558263:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1073 00000100:00080000:2.0:1789761146.558266:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761146.558269:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761146.558277:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761146.558288:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504005888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761146.558296:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761146.558299:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761146.558303:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761146.558304:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504005760:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761146.558308:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1073 00000100:00080000:2.0:1789761146.558315:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761146.558317:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761146.558320:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761146.558330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504006016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761146.558336:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761146.558339:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761146.558340:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761146.558343:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1073 00000100:00080000:2.0:1789761146.558345:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761146.558348:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761146.558351:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761146.558358:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504006144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761146.558365:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761146.558367:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789761146.558372:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761146.558372:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761146.558380:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1073 00000100:00080000:2.0:1789761146.558392:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761146.558395:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761146.558401:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761146.558410:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504006272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761146.558420:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761146.558421:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761146/real 1789761146] req@ffff9018d0f25500 x1876699504005760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761146.558428:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761146.558434:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761146.558442:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1073 00000100:00100000:1.0:1789761146.558446:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504005760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761146.558448:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761146.558458:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789761146.558460:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761146.558466:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:2.0:1789761146.558467:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761146.558471:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504005760:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761146.558482:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504006400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761146.558494:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504005888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761146.558513:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761146.558523:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761146/real 1789761146] req@ffff9018d0f26d80 x1876699504005888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761146.558531:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504005888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761146.558536:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761146.558538:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761146.558540:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504005888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761146.558544:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504006016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761146.558553:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761146.558559:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761146/real 1789761146] req@ffff9018d0f27800 x1876699504006016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761146.558566:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504006016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761146.558575:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761146.558578:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761146.558580:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504006016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761146.558583:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504006144:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761146.558593:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761146.558598:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761146/real 1789761146] req@ffff9018d0f25180 x1876699504006144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761146.558605:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504006144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761146.558611:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761146.558612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761146.558614:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504006144:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761146.558617:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504006272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761146.558627:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761146.558632:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761146/real 1789761146] req@ffff9018d0f24a80 x1876699504006272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761146.558639:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504006272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761146.558644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761146.558645:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761146.558648:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504006272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761146.558659:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504006400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761146.558669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761146.558675:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761146/real 1789761146] req@ffff9018d0f24380 x1876699504006400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761146.558682:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504006400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761146.558686:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761146.558688:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761146.558690:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504006400:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761147.198142:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761148.222242:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761149.246185:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761150.270180:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761151.294210:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761151.678128:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761151.678137:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761151.678140:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761151.678148:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1078 00000100:00080000:1.0:1789761151.678157:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761151.678162:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761151.678196:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504006528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761151.678225:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761151.678229:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761151.678231:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761151.678236:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1078 00000100:00080000:1.0:1789761151.678240:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761151.678245:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761151.678255:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761151.678270:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504006656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761151.678276:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504006528:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761151.678291:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761151.678294:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761151.678295:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761151.678299:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1078 00000100:00080000:1.0:1789761151.678302:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761151.678305:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761151.678309:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761151.678318:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504006784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761151.678325:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761151.678328:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761151.678329:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761151.678331:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1078 00000100:00080000:1.0:1789761151.678334:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761151.678336:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761151.678339:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789761151.678343:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761151.678350:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504006912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761151.678357:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761151.678361:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761151.678362:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761151.678365:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1078 00000100:00080000:1.0:1789761151.678367:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761151.678369:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761151.678372:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761151.678379:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504007040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761151.678384:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761151.678386:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761151.678387:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761151.678390:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1078 00000100:00080000:1.0:1789761151.678392:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761151.678394:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761151.678398:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761151.678403:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761151/real 1789761151] req@ffff9018c66a5180 x1876699504006528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761151.678406:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504007168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761151.678449:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504006528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761151.678460:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761151.678465:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761151.678469:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504006528:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761151.678493:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504006656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761151.678516:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761151.678530:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761151/real 1789761151] req@ffff9018c66a6680 x1876699504006656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761151.678542:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504006656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761151.678551:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761151.678555:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761151.678559:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504006656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761151.678565:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504006784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761151.678583:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761151.678593:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761151/real 1789761151] req@ffff9018c66a4700 x1876699504006784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761151.678606:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504006784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761151.678614:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761151.678618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761151.678621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504006784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761151.678628:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504006912:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761151.678647:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761151.678659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761151/real 1789761151] req@ffff9018c66a6300 x1876699504006912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761151.678671:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504006912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761151.678681:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761151.678684:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761151.678687:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504006912:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761151.678692:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504007040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761151.678709:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761151.678718:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761151/real 1789761151] req@ffff9018c66a4e00 x1876699504007040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761151.678729:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504007040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761151.678736:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761151.678739:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761151.678742:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504007040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761151.678747:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504007168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761151.678763:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761151.678772:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761151/real 1789761151] req@ffff9018c66a5500 x1876699504007168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761151.678782:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504007168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761151.678789:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761151.678792:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761151.678795:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504007168:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761152.318175:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761153.342256:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761154.366142:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761155.390233:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761156.414194:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761156.798213:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761156.798219:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761156.798222:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761156.798231:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1083 00000100:00080000:2.0:1789761156.798243:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761156.798249:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761156.798282:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504007296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761156.798311:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761156.798314:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761156.798315:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761156.798318:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1083 00000100:00080000:2.0:1789761156.798320:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761156.798323:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761156.798332:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761156.798345:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504007424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761156.798354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761156.798357:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761156.798358:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761156.798362:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1083 00000100:00100000:1.0:1789761156.798367:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504007296:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761156.798367:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761156.798375:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761156.798379:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761156.798391:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504007552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761156.798401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761156.798405:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761156.798408:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761156.798413:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1083 00000100:00080000:2.0:1789761156.798424:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761156.798435:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761156.798438:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761156.798445:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761156.798462:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504007680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761156.798470:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761156.798473:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761156.798474:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761156.798477:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1083 00000100:00080000:2.0:1789761156.798479:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761156.798483:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761156.798488:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761156/real 1789761156] req@ffff9018d0f26300 x1876699504007296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761156.798489:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761156.798501:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504007808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761156.798510:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761156.798514:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761156.798516:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761156.798518:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504007296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761156.798522:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1083 00000100:00080000:2.0:1789761156.798527:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761156.798533:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761156.798538:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761156.798550:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504007936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761156.798553:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761156.798565:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761156.798571:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504007296:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761156.798582:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504007424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761156.798613:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761156.798631:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761156/real 1789761156] req@ffff9018d0f27100 x1876699504007424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761156.798648:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504007424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761156.798662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761156.798666:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761156.798669:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504007424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761156.798691:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504007552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761156.798716:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761156.798730:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761156/real 1789761156] req@ffff9018d0f26a00 x1876699504007552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761156.798744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504007552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761156.798754:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761156.798758:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761156.798762:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504007552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761156.798779:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504007680:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761156.798812:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761156.798828:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761156/real 1789761156] req@ffff9018d0f25c00 x1876699504007680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761156.798846:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504007680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761156.798868:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761156.798875:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761156.798886:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504007680:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761156.798901:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504007808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761156.798934:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761156.798947:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761156/real 1789761156] req@ffff9018d0f25f80 x1876699504007808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761156.798959:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504007808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761156.798968:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761156.798971:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761156.798974:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504007808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761156.798981:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504007936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761156.799000:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761156.799012:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761156/real 1789761156] req@ffff9018d0f24380 x1876699504007936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761156.799024:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504007936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761156.799064:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761156.799069:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761156.799074:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504007936:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761157.438222:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761158.462277:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761159.486185:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761160.510200:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761161.534194:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761161.918141:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761161.918152:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761161.918156:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761161.918167:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1088 00000100:00080000:1.0:1789761161.918178:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761161.918185:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761161.918234:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504008064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761161.918264:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761161.918267:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761161.918268:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761161.918272:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1088 00000100:00080000:1.0:1789761161.918275:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761161.918280:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761161.918290:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761161.918301:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504008192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761161.918313:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504008064:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761161.918339:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761161.918342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761161.918344:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761161.918348:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1088 00000100:00080000:1.0:1789761161.918351:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761161.918355:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761161.918359:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761161.918368:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504008320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761161.918375:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761161.918378:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761161.918379:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761161.918383:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1088 00000100:00080000:1.0:1789761161.918385:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761161.918393:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761161.918397:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761161.918404:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504008448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761161.918412:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761161.918415:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761161.918416:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761161.918422:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1088 00000100:00080000:1.0:1789761161.918429:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761161.918433:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761161.918438:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761161.918446:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504008192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761161.918455:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504008576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761161.918472:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761161.918477:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761161.918479:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761161.918484:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504008320:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789761161.918488:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1088 00000100:00080000:1.0:1789761161.918497:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761161.918504:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761161.918512:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761161.918513:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504008448:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761161.918535:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504008704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761161.918565:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504008576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761161.918588:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504008704:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761162.558210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761163.582206:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761164.606218:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761164.990389:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:0.0:1789761164.990451:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761164.990472:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789761164.990525:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789761164.990537:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789761164.990550:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789761164.990560:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789761164.990573:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:2.0:1789761164.990617:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761161/real 1789761164] req@ffff9018c66a6d80 x1876699504008576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761164.990655:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504008576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761164.990668:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761164.990673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761164.990679:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504008576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761164.990688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761161/real 1789761164] req@ffff9018c66a5c00 x1876699504008704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761164.990700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504008704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761164.990708:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761164.990711:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761164.990714:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504008704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761164.990722:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761161/real 1789761164] req@ffff9018c66a7b80 x1876699504008192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761164.990734:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504008192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761164.990740:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761164.990743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761164.990746:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504008192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761164.990753:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761161/real 1789761164] req@ffff9018c66a4000 x1876699504008320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761164.990765:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504008320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761164.990772:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761164.990775:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761164.990779:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504008320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761164.990786:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761161/real 1789761164] req@ffff9018c66a5f80 x1876699504008448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761164.990845:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504008448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761164.990855:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761164.990859:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761164.990863:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504008448:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761164.990871:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761161/real 1789761164] req@ffff9018c66a7800 x1876699504008064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761164.990899:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504008064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761164.990907:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761164.990911:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761164.990914:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504008064:192.168.203.158@tcp:250:kworker.0 00010000:00010000:3.0:1789761165.630185:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761166.654261:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761167.038444:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761167.038460:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761167.038466:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761167.038479:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1093 00000100:00080000:0.0:1789761167.038495:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761167.038505:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761167.038553:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504008832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761167.038595:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761167.038600:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761167.038602:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761167.038607:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1093 00000100:00080000:0.0:1789761167.038612:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761167.038616:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761167.038630:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761167.038644:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504008960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761167.038655:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761167.038659:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761167.038661:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761167.038665:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1093 00000100:00080000:0.0:1789761167.038669:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761167.038673:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761167.038678:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761167.038688:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504009088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761167.038705:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761167.038710:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761167.038712:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761167.038716:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1093 00000100:00080000:0.0:1789761167.038720:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761167.038724:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761167.038729:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761167.038741:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504009216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761167.038750:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761167.038754:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761167.038756:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761167.038759:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1093 00000100:00080000:0.0:1789761167.038763:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761167.038767:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761167.038772:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761167.038796:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504009344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761167.038807:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761167.038811:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761167.038813:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761167.038817:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1093 00000100:00080000:0.0:1789761167.038821:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761167.038826:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761167.038831:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761167.038842:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504009472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761167.038948:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504008832:192.168.203.158@tcp:250:kworker.0 00000800:00000100:2.0:1789761167.039080:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761167.039137:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761167/real 1789761167] req@ffff9018c8b9ad80 x1876699504008832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761167.039159:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504008832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761167.039180:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761167.039186:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761167.039191:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504008832:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761167.039200:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504008960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761167.039227:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761167.039242:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761167/real 1789761167] req@ffff9018c8b99500 x1876699504008960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761167.039255:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504008960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761167.039262:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761167.039264:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761167.039266:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504008960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761167.039271:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504009088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761167.039292:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761167.039303:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761167/real 1789761167] req@ffff9018c8b99c00 x1876699504009088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761167.039314:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504009088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761167.039323:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761167.039327:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761167.039331:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504009088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761167.039337:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504009216:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761167.039365:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761167.039379:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761167/real 1789761167] req@ffff9018c8b98380 x1876699504009216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761167.039393:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504009216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761167.039401:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761167.039404:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761167.039407:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504009216:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761167.039413:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504009344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761167.039434:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761167.039446:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761167/real 1789761167] req@ffff9018c8b9a300 x1876699504009344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761167.039460:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504009344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761167.039468:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761167.039472:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761167.039475:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504009344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761167.039482:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504009472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761167.039503:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761167.039514:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761167/real 1789761167] req@ffff9018c8b9b100 x1876699504009472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761167.039526:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504009472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761167.039534:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761167.039537:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761167.039540:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504009472:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761167.678225:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761168.702226:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761169.726214:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761170.750244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761171.774162:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761172.158127:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761172.158135:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761172.158138:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761172.158147:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1098 00000100:00080000:0.0:1789761172.158156:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761172.158162:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761172.158192:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504009600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761172.158214:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761172.158216:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761172.158218:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761172.158220:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1098 00000100:00080000:0.0:1789761172.158222:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761172.158225:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761172.158233:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761172.158241:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504009728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761172.158249:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761172.158259:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504009600:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761172.158260:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761172.158264:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761172.158269:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1098 00000100:00080000:0.0:1789761172.158272:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761172.158274:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761172.158278:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761172.158286:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504009856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761172.158293:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761172.158296:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761172.158297:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761172.158299:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1098 00000100:00080000:0.0:1789761172.158302:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761172.158304:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761172.158307:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761172.158314:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504009984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761172.158317:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761172.158327:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761172.158330:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761172.158332:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761172.158336:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1098 00000100:00080000:0.0:1789761172.158338:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761172.158341:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761172.158345:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761172.158352:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761172/real 1789761172] req@ffff9018c8b9a680 x1876699504009600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761172.158355:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504010112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761172.158380:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761172.158383:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789761172.158385:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504009600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761172.158391:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761172.158399:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761172.158400:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1098 00000100:00080000:0.0:1789761172.158406:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761172.158408:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761172.158413:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761172.158416:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504009600:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789761172.158418:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761172.158429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504010240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761172.158441:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504009728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761172.158458:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761172.158476:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761172/real 1789761172] req@ffff9018c8b98000 x1876699504009728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761172.158488:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504009728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761172.158499:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761172.158502:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761172.158505:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504009728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761172.158511:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504009856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761172.158524:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761172.158530:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761172/real 1789761172] req@ffff9018c8b9aa00 x1876699504009856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761172.158537:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504009856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761172.158542:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761172.158544:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761172.158546:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504009856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761172.158550:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504009984:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761172.158560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761172.158566:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761172/real 1789761172] req@ffff9018c8b9b800 x1876699504009984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761172.158573:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504009984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761172.158578:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761172.158579:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761172.158582:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504009984:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761172.158585:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504010112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761172.158595:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761172.158600:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761172/real 1789761172] req@ffff9018c8b9b480 x1876699504010112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761172.158606:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504010112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761172.158611:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761172.158613:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761172.158615:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504010112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761172.158618:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504010240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761172.158627:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761172.158632:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761172/real 1789761172] req@ffff9018c8b99f80 x1876699504010240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761172.158638:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504010240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761172.158642:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761172.158644:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761172.158646:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504010240:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761172.798187:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761173.822182:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761174.846170:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761175.870187:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761176.894176:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761177.278103:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761177.278112:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761177.278115:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761177.278124:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1103 00000100:00080000:0.0:1789761177.278133:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761177.278138:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761177.278170:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504010368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761177.278190:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761177.278193:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761177.278194:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761177.278197:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1103 00000100:00080000:0.0:1789761177.278200:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761177.278203:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761177.278211:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761177.278219:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504010496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761177.278227:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761177.278229:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761177.278230:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761177.278233:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1103 00000100:00080000:0.0:1789761177.278237:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761177.278238:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504010368:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761177.278240:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761177.278245:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761177.278253:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504010624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761177.278258:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761177.278260:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761177.278261:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761177.278264:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1103 00000100:00080000:0.0:1789761177.278265:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761177.278268:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761177.278272:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761177.278280:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504010752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761177.278291:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761177.278294:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761177.278295:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761177.278297:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1103 00000100:00080000:0.0:1789761177.278299:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761177.278302:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761177.278305:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761177.278311:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504010880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761177.278317:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761177.278321:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761177.278325:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761177.278326:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761177.278330:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1103 00000100:00080000:0.0:1789761177.278333:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761177.278335:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761177.278338:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761177.278345:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504011008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761177.278375:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761177/real 1789761177] req@ffff9018c8b99880 x1876699504010368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761177.278398:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504010368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761177.278411:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761177.278417:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761177.278420:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504010368:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761177.278440:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504010496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761177.278458:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761177.278467:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761177/real 1789761177] req@ffff9018c8b98e00 x1876699504010496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761177.278475:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504010496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761177.278481:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761177.278483:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761177.278485:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504010496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761177.278488:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504010624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761177.278500:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761177.278509:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761177/real 1789761177] req@ffff9018c8b98a80 x1876699504010624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761177.278522:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504010624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761177.278537:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761177.278541:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761177.278545:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504010624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761177.278551:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504010752:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761177.278568:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761177.278576:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761177/real 1789761177] req@ffff9018c8b98700 x1876699504010752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761177.278584:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504010752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761177.278589:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761177.278591:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761177.278593:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504010752:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761177.278597:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504010880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761177.278607:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761177.278613:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761177/real 1789761177] req@ffff9018c8b9bb80 x1876699504010880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761177.278620:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504010880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761177.278624:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761177.278626:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761177.278628:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504010880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761177.278632:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504011008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761177.278642:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761177.278647:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761177/real 1789761177] req@ffff9018c8b99f80 x1876699504011008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761177.278654:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504011008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761177.278659:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761177.278661:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761177.278662:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504011008:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761177.918171:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761178.942254:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761179.966167:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761180.990221:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761182.014222:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761182.398153:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761182.398164:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761182.398173:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761182.398185:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1108 00000100:00080000:2.0:1789761182.398196:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761182.398204:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761182.398239:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504011136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761182.398268:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761182.398272:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761182.398274:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761182.398278:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1108 00000100:00080000:2.0:1789761182.398281:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761182.398286:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761182.398297:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761182.398318:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504011264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761182.398323:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504011136:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761182.398334:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761182.398338:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761182.398342:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761182.398348:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1108 00000100:00080000:2.0:1789761182.398352:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761182.398355:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761182.398360:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761182.398371:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504011392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761182.398381:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761182.398384:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761182.398386:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761182.398391:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1108 00000100:00080000:2.0:1789761182.398395:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761182.398398:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761182.398403:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789761182.398405:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761182.398417:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504011520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761182.398427:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761182.398431:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761182.398433:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761182.398437:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1108 00000100:00080000:2.0:1789761182.398441:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761182.398445:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761182.398450:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761182.398462:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761182/real 1789761182] req@ffff9018d0f24a80 x1876699504011136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761182.398463:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504011648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761182.398493:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761182.398498:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761182.398500:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761182.398507:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1108 00000100:00100000:1.0:1789761182.398508:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504011136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761182.398511:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761182.398520:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789761182.398524:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789761182.398527:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761182.398531:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761182.398536:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504011136:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761182.398542:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504011776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761182.398561:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504011264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761182.398584:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761182.398601:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761182/real 1789761182] req@ffff9018d0f25180 x1876699504011264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761182.398610:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504011264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761182.398616:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761182.398619:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761182.398621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504011264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761182.398627:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504011392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761182.398647:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761182.398659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761182/real 1789761182] req@ffff9018d0f27800 x1876699504011392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761182.398685:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504011392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761182.398695:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761182.398698:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761182.398700:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504011392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761182.398705:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504011520:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761182.398721:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761182.398730:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761182/real 1789761182] req@ffff9018d0f26d80 x1876699504011520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761182.398738:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504011520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761182.398743:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761182.398745:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761182.398747:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504011520:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761182.398751:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504011648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761182.398762:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761182.398768:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761182/real 1789761182] req@ffff9018d0f25500 x1876699504011648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761182.398775:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504011648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761182.398780:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761182.398782:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761182.398784:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504011648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761182.398787:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504011776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761182.398797:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761182.398802:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761182/real 1789761182] req@ffff9018d0f24000 x1876699504011776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761182.398809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504011776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761182.398819:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761182.398821:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761182.398823:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504011776:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761183.038149:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761184.062199:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761185.086176:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761186.110205:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761187.134166:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761187.518155:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761187.518163:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761187.518166:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761187.518173:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1114 00000100:00080000:1.0:1789761187.518182:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761187.518187:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761187.518224:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504011904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761187.518243:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761187.518246:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761187.518248:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761187.518250:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1114 00000100:00080000:1.0:1789761187.518253:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761187.518256:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761187.518263:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761187.518272:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504012032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761187.518279:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504011904:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761187.518283:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761187.518287:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761187.518288:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761187.518291:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1114 00000100:00080000:1.0:1789761187.518293:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761187.518295:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761187.518299:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761187.518307:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504012160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761187.518313:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761187.518315:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761187.518317:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761187.518320:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1114 00000100:00080000:1.0:1789761187.518324:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761187.518328:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761187.518334:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789761187.518338:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761187.518351:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504012288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761187.518365:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761187.518368:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761187.518370:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761187.518374:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1114 00000100:00100000:2.0:1789761187.518379:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761187/real 1789761187] req@ffff9018c66a7800 x1876699504011904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761187.518381:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761187.518391:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761187.518398:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761187.518401:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504011904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761187.518415:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504012416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761187.518418:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761187.518426:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761187.518433:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504011904:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761187.518435:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761187.518440:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761187.518441:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761187.518445:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1114 00000100:00080000:1.0:1789761187.518449:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761187.518451:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761187.518456:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761187.518457:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504012032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761187.518469:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504012544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761187.518489:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761187.518506:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761187/real 1789761187] req@ffff9018c66a5f80 x1876699504012032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761187.518521:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504012032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761187.518527:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761187.518530:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761187.518532:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504012032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761187.518537:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504012160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761187.518552:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761187.518560:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761187/real 1789761187] req@ffff9018c66a4000 x1876699504012160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761187.518567:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504012160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761187.518573:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761187.518575:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761187.518577:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504012160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761187.518581:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504012288:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761187.518592:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761187.518598:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761187/real 1789761187] req@ffff9018c66a7b80 x1876699504012288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761187.518605:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504012288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761187.518611:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761187.518612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761187.518614:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504012288:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761187.518618:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504012416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761187.518640:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761187.518649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761187/real 1789761187] req@ffff9018c66a5c00 x1876699504012416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761187.518657:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504012416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761187.518662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761187.518665:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761187.518667:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504012416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761187.518681:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504012544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761187.518695:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761187.518701:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761187/real 1789761187] req@ffff9018c66a6d80 x1876699504012544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761187.518709:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504012544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761187.518714:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761187.518716:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761187.518718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504012544:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761188.158215:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761189.182206:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761190.206223:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761191.230158:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761192.254216:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761192.638148:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761192.638159:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761192.638162:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761192.638171:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1119 00000100:00080000:1.0:1789761192.638180:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761192.638186:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761192.638217:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504012672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761192.638240:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761192.638243:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761192.638245:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761192.638249:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1119 00000100:00080000:1.0:1789761192.638252:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761192.638255:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761192.638263:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761192.638275:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504012800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761192.638278:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504012672:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761192.638293:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761192.638295:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761192.638297:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761192.638301:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1119 00000100:00080000:1.0:1789761192.638304:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761192.638307:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761192.638311:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761192.638321:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504012928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761192.638329:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761192.638333:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761192.638335:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761192.638340:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761192.638344:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1119 00000100:00080000:1.0:1789761192.638351:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761192.638356:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761192.638362:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761192.638377:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504013056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761192.638388:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761192/real 1789761192] req@ffff9018c66a5500 x1876699504012672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761192.638388:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761192.638398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761192.638402:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761192.638412:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1119 00000100:00080000:1.0:1789761192.638420:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761192.638427:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761192.638436:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504012672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789761192.638438:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761192.638450:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789761192.638455:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504013184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761192.638456:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761192.638462:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504012672:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761192.638475:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761192.638478:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761192.638479:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761192.638483:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1119 00000100:00080000:1.0:1789761192.638487:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761192.638490:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504012800:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789761192.638493:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761192.638499:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761192.638509:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504013312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761192.638518:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761192.638532:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761192/real 1789761192] req@ffff9018c66a4e00 x1876699504012800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761192.638541:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504012800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761192.638547:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761192.638549:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761192.638551:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504012800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761192.638555:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504012928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761192.638570:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761192.638580:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761192/real 1789761192] req@ffff9018c66a6300 x1876699504012928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761192.638609:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504012928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761192.638625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761192.638629:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761192.638633:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504012928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761192.638639:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504013056:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761192.638663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761192.638677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761192/real 1789761192] req@ffff9018c66a4700 x1876699504013056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761192.638687:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504013056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761192.638694:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761192.638696:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761192.638698:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504013056:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761192.638704:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504013184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761192.638721:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761192.638729:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761192/real 1789761192] req@ffff9018c66a6680 x1876699504013184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761192.638741:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504013184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761192.638748:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761192.638752:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761192.638756:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504013184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761192.638783:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504013312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761192.638795:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761192.638802:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761192/real 1789761192] req@ffff9018c66a5180 x1876699504013312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761192.638809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504013312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761192.638814:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761192.638816:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761192.638818:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504013312:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761193.278155:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761194.302180:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761195.326222:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761196.350202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761197.374159:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761197.758152:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761197.758160:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761197.758164:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761197.758172:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1124 00000100:00080000:0.0:1789761197.758180:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761197.758186:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761197.758218:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504013440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761197.758242:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761197.758245:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761197.758246:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761197.758249:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1124 00000100:00080000:0.0:1789761197.758251:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761197.758254:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761197.758262:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761197.758271:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504013568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761197.758278:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761197.758281:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761197.758283:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761197.758285:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1124 00000100:00080000:0.0:1789761197.758287:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761197.758290:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761197.758293:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761197.758296:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504013440:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761197.758301:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504013696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761197.758309:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761197.758311:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761197.758313:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761197.758316:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1124 00000100:00080000:0.0:1789761197.758318:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761197.758320:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761197.758323:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761197.758332:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504013824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761197.758339:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761197.758342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761197.758343:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761197.758346:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1124 00000100:00080000:0.0:1789761197.758353:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761197.758358:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789761197.758359:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789761197.758364:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761197.758373:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504013952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761197.758379:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761197.758381:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761197.758383:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761197.758385:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1124 00000100:00080000:0.0:1789761197.758388:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761197.758391:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761197.758394:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761197.758400:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504014080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761197.758405:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761197/real 1789761197] req@ffff9018c8b9b480 x1876699504013440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761197.758423:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504013440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761197.758432:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761197.758435:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761197.758439:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504013440:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761197.758445:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504013568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761197.758460:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761197.758477:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761197/real 1789761197] req@ffff9018c8b9b800 x1876699504013568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761197.758486:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504013568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761197.758496:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761197.758498:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761197.758501:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504013568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761197.758518:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504013696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761197.758530:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761197.758537:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761197/real 1789761197] req@ffff9018c8b9aa00 x1876699504013696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761197.758544:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504013696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761197.758549:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761197.758564:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761197.758566:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504013696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761197.758571:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504013824:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761197.758588:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761197.758596:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761197/real 1789761197] req@ffff9018c8b98000 x1876699504013824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761197.758603:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504013824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761197.758609:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761197.758611:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761197.758613:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504013824:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761197.758617:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504013952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761197.758628:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761197.758634:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761197/real 1789761197] req@ffff9018c8b9a680 x1876699504013952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761197.758641:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504013952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761197.758645:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761197.758647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761197.758649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504013952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761197.758652:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504014080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761197.758662:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761197.758667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761197/real 1789761197] req@ffff9018c8b9b100 x1876699504014080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761197.758674:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504014080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761197.758679:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761197.758681:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761197.758683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504014080:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761198.398187:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761199.422195:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761200.446221:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761201.470203:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761202.494171:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761202.878108:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761202.878117:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761202.878120:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761202.878128:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1129 00000100:00080000:1.0:1789761202.878137:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761202.878143:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761202.878174:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504014208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761202.878192:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761202.878195:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761202.878196:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761202.878199:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1129 00000100:00080000:1.0:1789761202.878202:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761202.878205:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761202.878213:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761202.878222:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504014336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761202.878229:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761202.878233:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789761202.878234:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504014208:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761202.878235:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761202.878243:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1129 00000100:00080000:1.0:1789761202.878247:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761202.878250:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761202.878253:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761202.878262:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504014464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761202.878268:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761202.878271:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761202.878272:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761202.878274:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1129 00000100:00080000:1.0:1789761202.878277:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761202.878280:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761202.878284:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761202.878291:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504014592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761202.878297:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761202.878299:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761202.878301:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789761202.878305:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761202.878306:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1129 00000100:00080000:1.0:1789761202.878311:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761202.878314:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761202.878318:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761202.878325:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504014720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761202.878331:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761202.878334:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761202.878335:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761202.878338:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1129 00000100:00080000:1.0:1789761202.878340:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761202.878342:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761202.878346:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761202.878352:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761202/real 1789761202] req@ffff9018c66a4380 x1876699504014208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761202.878361:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504014848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761202.878377:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504014208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761202.878403:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761202.878408:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761202.878411:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504014208:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761202.878434:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504014336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761202.878452:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761202.878461:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761202/real 1789761202] req@ffff9018c66a7100 x1876699504014336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761202.878471:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504014336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761202.878480:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761202.878483:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761202.878487:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504014336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761202.878493:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504014464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761202.878528:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761202.878542:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761202/real 1789761202] req@ffff9018c66a7480 x1876699504014464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761202.878556:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504014464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761202.878573:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761202.878577:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761202.878581:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504014464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761202.878587:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504014592:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761202.878608:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761202.878620:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761202/real 1789761202] req@ffff9018c66a4a80 x1876699504014592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761202.878633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504014592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761202.878642:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761202.878646:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761202.878650:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504014592:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761202.878656:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504014720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761202.878674:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761202.878685:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761202/real 1789761202] req@ffff9018c66a6a00 x1876699504014720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761202.878697:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504014720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761202.878706:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761202.878710:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761202.878714:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504014720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761202.878720:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504014848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761202.878739:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761202.878748:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761202/real 1789761202] req@ffff9018c66a5880 x1876699504014848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761202.878755:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504014848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761202.878761:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761202.878763:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761202.878765:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504014848:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761203.518170:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761204.542172:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761205.566188:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761206.590163:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761207.614168:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761207.998118:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761207.998126:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761207.998130:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761207.998138:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1134 00000100:00080000:0.0:1789761207.998147:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761207.998153:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761207.998184:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504014976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761207.998210:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761207.998213:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761207.998215:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761207.998218:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1134 00000100:00080000:0.0:1789761207.998220:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761207.998224:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761207.998232:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761207.998243:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504015104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761207.998245:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504014976:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761207.998259:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761207.998261:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761207.998262:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761207.998265:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1134 00000100:00080000:0.0:1789761207.998267:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761207.998269:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761207.998272:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761207.998281:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504015232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761207.998287:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761207.998289:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761207.998291:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761207.998293:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1134 00000100:00080000:0.0:1789761207.998296:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761207.998299:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761207.998302:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789761207.998311:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761207.998316:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504015360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761207.998328:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761207.998331:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761207.998334:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761207.998339:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1134 00000100:00080000:0.0:1789761207.998344:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761207.998345:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761207/real 1789761207] req@ffff9018c8b9a300 x1876699504014976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761207.998353:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761207.998362:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761207.998366:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504014976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761207.998376:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789761207.998377:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504015488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761207.998382:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761207.998389:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504014976:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761207.998395:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761207.998398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761207.998400:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761207.998404:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1134 00000100:00080000:0.0:1789761207.998407:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761207.998410:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761207.998414:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504015104:192.168.203.158@tcp:38:kworker.0 00000008:00100000:0.0:1789761207.998415:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761207.998430:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504015616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761207.998441:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761207.998456:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761207/real 1789761207] req@ffff9018c8b98380 x1876699504015104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761207.998468:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504015104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761207.998487:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761207.998491:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761207.998493:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504015104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761207.998498:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504015232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761207.998519:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761207.998527:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761207/real 1789761207] req@ffff9018c8b99c00 x1876699504015232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761207.998535:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504015232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761207.998540:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761207.998542:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761207.998544:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504015232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761207.998548:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504015360:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761207.998559:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761207.998565:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761207/real 1789761207] req@ffff9018c8b99500 x1876699504015360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761207.998572:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504015360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761207.998578:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761207.998580:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761207.998582:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504015360:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761207.998585:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504015488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761207.998594:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761207.998600:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761207/real 1789761207] req@ffff9018c8b9ad80 x1876699504015488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761207.998607:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504015488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761207.998612:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761207.998614:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761207.998615:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504015488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761207.998628:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504015616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761207.998638:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761207.998644:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761207/real 1789761207] req@ffff9018c8b9a300 x1876699504015616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761207.998650:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504015616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761207.998655:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761207.998657:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761207.998659:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504015616:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761208.638149:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761209.662231:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761210.686179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761211.710198:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761212.734167:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761213.118167:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761213.118178:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761213.118182:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761213.118194:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1139 00000100:00080000:0.0:1789761213.118206:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761213.118212:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761213.118248:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504015744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761213.118276:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761213.118280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761213.118282:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761213.118286:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1139 00000100:00080000:0.0:1789761213.118290:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761213.118295:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761213.118306:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761213.118313:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504015744:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761213.118321:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504015872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761213.118334:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761213.118338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761213.118340:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761213.118344:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1139 00000100:00080000:0.0:1789761213.118348:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761213.118352:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761213.118358:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761213.118371:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504016000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761213.118379:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761213.118388:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761213.118392:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761213.118394:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761213.118399:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1139 00000100:00080000:0.0:1789761213.118404:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761213.118408:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761213.118413:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761213.118425:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504016128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761213.118429:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761213/real 1789761213] req@ffff9018c8b9b100 x1876699504015744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761213.118464:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761213.118478:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761213.118481:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761213.118485:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504015744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761213.118490:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1139 00000100:00080000:0.0:1789761213.118498:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761213.118506:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789761213.118508:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789761213.118513:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789761213.118517:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761213.118524:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504015744:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761213.118530:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504016256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761213.118540:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761213.118543:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761213.118546:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761213.118548:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504015872:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761213.118553:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1139 00000100:00080000:0.0:1789761213.118558:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761213.118562:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761213.118567:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761213.118576:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761213.118581:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504016384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761213.118591:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761213/real 1789761213] req@ffff9018c8b9a680 x1876699504015872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761213.118603:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504015872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761213.118610:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761213.118612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761213.118615:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504015872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761213.118620:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504016000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761213.118636:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761213.118648:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761213/real 1789761213] req@ffff9018c8b98000 x1876699504016000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761213.118656:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504016000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761213.118662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761213.118664:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761213.118668:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504016000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761213.118672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504016128:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761213.118683:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761213.118688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761213/real 1789761213] req@ffff9018c8b9aa00 x1876699504016128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761213.118696:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504016128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761213.118701:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761213.118702:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761213.118704:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504016128:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761213.118707:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504016256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761213.118718:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761213.118724:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761213/real 1789761213] req@ffff9018c8b9b800 x1876699504016256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761213.118731:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504016256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761213.118736:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761213.118738:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761213.118740:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504016256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761213.118755:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504016384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761213.118773:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761213.118784:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761213/real 1789761213] req@ffff9018c8b9b480 x1876699504016384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761213.118797:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504016384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761213.118804:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761213.118807:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761213.118809:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504016384:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761213.758214:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761214.782157:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761215.806203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761216.830158:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761217.854207:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761218.238135:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761218.238147:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761218.238151:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761218.238161:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1144 00000100:00080000:0.0:1789761218.238171:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761218.238180:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761218.238230:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504016512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761218.238251:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761218.238254:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761218.238255:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761218.238258:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1144 00000100:00080000:0.0:1789761218.238261:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761218.238268:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761218.238277:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761218.238291:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504016640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761218.238291:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504016512:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761218.238311:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761218.238315:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761218.238317:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761218.238321:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1144 00000100:00080000:0.0:1789761218.238326:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761218.238330:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761218.238343:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761218.238356:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761218.238359:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504016768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761218.238371:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761218.238374:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761218.238376:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761218.238379:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1144 00000100:00080000:0.0:1789761218.238383:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761218.238386:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761218.238392:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761218.238397:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761218/real 1789761218] req@ffff9018c8b99f80 x1876699504016512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761218.238419:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504016896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761218.238431:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761218.238433:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761218.238435:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761218.238438:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504016512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761218.238446:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1144 00000100:00080000:0.0:1789761218.238452:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761218.238458:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789761218.238461:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789761218.238462:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789761218.238472:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761218.238476:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504017024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761218.238482:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504016512:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761218.238486:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761218.238492:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761218.238493:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761218.238497:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1144 00000100:00080000:0.0:1789761218.238499:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761218.238501:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761218.238504:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761218.238510:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504017152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761218.238516:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504016640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761218.238542:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761218.238555:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761218/real 1789761218] req@ffff9018c8b9bb80 x1876699504016640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761218.238567:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504016640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761218.238577:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761218.238580:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761218.238584:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504016640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761218.238590:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504016768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761218.238610:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761218.238622:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761218/real 1789761218] req@ffff9018c8b98700 x1876699504016768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761218.238635:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504016768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761218.238644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761218.238647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761218.238651:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504016768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761218.238657:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504016896:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761218.238676:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761218.238687:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761218/real 1789761218] req@ffff9018c8b98a80 x1876699504016896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761218.238699:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504016896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761218.238707:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761218.238710:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761218.238713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504016896:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761218.238719:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504017024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761218.238732:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761218.238741:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761218/real 1789761218] req@ffff9018c8b98e00 x1876699504017024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761218.238752:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504017024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761218.238760:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761218.238763:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761218.238766:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504017024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761218.238782:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504017152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761218.238797:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761218.238806:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761218/real 1789761218] req@ffff9018c8b99880 x1876699504017152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761218.238817:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504017152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761218.238825:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761218.238829:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761218.238832:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504017152:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761218.878208:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761219.902185:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761220.926148:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761221.950228:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761222.974179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761223.358126:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761223.358135:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761223.358138:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761223.358146:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1149 00000100:00080000:2.0:1789761223.358155:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761223.358161:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761223.358193:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504017280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761223.358218:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761223.358221:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761223.358223:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761223.358226:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1149 00000100:00080000:2.0:1789761223.358228:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761223.358231:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761223.358239:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761223.358248:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504017408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761223.358253:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504017280:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761223.358261:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761223.358264:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761223.358266:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761223.358269:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1149 00000100:00080000:2.0:1789761223.358271:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761223.358273:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761223.358276:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761223.358284:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504017536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761223.358291:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761223.358293:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761223.358295:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761223.358297:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1149 00000100:00080000:2.0:1789761223.358299:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761223.358301:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761223.358305:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761223.358312:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504017664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761223.358319:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761223.358326:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761223.358329:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761223.358330:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761223.358334:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1149 00000100:00080000:2.0:1789761223.358339:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761223.358341:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761223.358344:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761223.358354:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504017792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761223.358377:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761223/real 1789761223] req@ffff9018d0f27480 x1876699504017280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761223.358378:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761223.358387:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761223.358390:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761223.358398:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1149 00000100:00080000:2.0:1789761223.358404:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761223.358408:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504017280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761223.358413:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789761223.358423:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789761223.358424:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761223.358429:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761223.358434:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504017280:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761223.358443:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504017920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761223.358462:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504017408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761223.358483:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761223.358492:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761223/real 1789761223] req@ffff9018d0f24e00 x1876699504017408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761223.358501:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504017408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761223.358506:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761223.358508:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761223.358511:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504017408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761223.358514:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504017536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761223.358525:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761223.358531:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761223/real 1789761223] req@ffff9018d0f26680 x1876699504017536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761223.358538:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504017536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761223.358543:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761223.358545:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761223.358546:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504017536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761223.358550:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504017664:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761223.358560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761223.358566:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761223/real 1789761223] req@ffff9018d0f24700 x1876699504017664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761223.358573:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504017664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761223.358579:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761223.358580:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761223.358582:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504017664:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761223.358585:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504017792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761223.358594:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761223.358600:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761223/real 1789761223] req@ffff9018d0f24000 x1876699504017792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761223.358607:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504017792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761223.358612:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761223.358614:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761223.358615:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504017792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761223.358619:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504017920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761223.358628:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761223.358633:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761223/real 1789761223] req@ffff9018d0f25500 x1876699504017920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761223.358641:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504017920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761223.358654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761223.358658:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761223.358661:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504017920:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761223.998184:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761225.022195:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761226.046181:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761227.070183:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761228.094181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761228.478130:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761228.478139:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761228.478143:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761228.478152:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1155 00000100:00080000:2.0:1789761228.478161:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761228.478168:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761228.478197:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504018048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761228.478222:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761228.478225:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761228.478226:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761228.478230:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1155 00000100:00080000:2.0:1789761228.478234:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761228.478240:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761228.478251:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761228.478261:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504018048:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761228.478269:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504018176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761228.478292:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761228.478297:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761228.478299:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761228.478305:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1155 00000100:00080000:2.0:1789761228.478312:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761228.478317:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761228.478339:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761228.478355:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504018304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761228.478367:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504018176:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761228.478369:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761228.478376:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761228.478379:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761228.478384:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1155 00000100:00080000:2.0:1789761228.478389:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761228.478393:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761228.478400:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761228.478404:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504018304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761228.478415:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504018432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761228.478428:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761228.478432:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761228.478434:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761228.478437:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504018432:192.168.203.158@tcp:8:kworker.0 00000100:00080000:2.0:1789761228.478443:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1155 00000100:00080000:2.0:1789761228.478449:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761228.478454:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761228.478459:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761228.478475:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504018560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761228.478494:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761228.478499:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761228.478502:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761228.478507:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1155 00000100:00080000:2.0:1789761228.478511:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761228.478516:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761228.478522:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761228.478524:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504018560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761228.478538:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504018688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761228.478578:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504018688:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761229.118191:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761230.142173:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761231.166213:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789761231.550329:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789761231.550393:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789761231.550410:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761231.550462:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761231.550472:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761231.550481:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761231.550492:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761231.550505:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789761231.550516:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761228/real 1789761231] req@ffff9018d0f24380 x1876699504018560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761231.550544:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504018560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761231.550558:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761231.550564:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761231.550569:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504018560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761231.550578:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761228/real 1789761231] req@ffff9018d0f24a80 x1876699504018432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761231.550593:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504018432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761231.550602:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761231.550605:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761231.550609:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504018432:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761231.550616:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761228/real 1789761231] req@ffff9018d0f25180 x1876699504018304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761231.550630:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504018304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761231.550638:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761231.550642:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761231.550652:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504018304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761231.550659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761228/real 1789761231] req@ffff9018d0f27800 x1876699504018176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761231.550673:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504018176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761231.550681:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761231.550684:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761231.550687:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504018176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761231.550693:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761228/real 1789761231] req@ffff9018d0f26d80 x1876699504018048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761231.550706:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504018048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761231.550713:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761231.550717:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761231.550720:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504018048:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761231.550749:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761228/real 1789761231] req@ffff9018d0f25f80 x1876699504018688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761231.550761:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504018688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761231.550768:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761231.550771:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761231.550774:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504018688:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761232.190211:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761233.214221:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761233.598159:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761233.598167:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761233.598170:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761233.598178:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1160 00000100:00080000:1.0:1789761233.598187:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761233.598192:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761233.598219:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504018816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761233.598242:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761233.598245:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761233.598246:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761233.598249:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1160 00000100:00080000:1.0:1789761233.598252:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761233.598254:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761233.598261:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761233.598270:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504018944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761233.598278:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761233.598280:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761233.598282:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761233.598300:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1160 00000100:00080000:1.0:1789761233.598303:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761233.598309:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761233.598311:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504018816:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789761233.598314:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761233.598330:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504019072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761233.598341:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761233.598345:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761233.598347:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761233.598351:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1160 00000100:00080000:1.0:1789761233.598363:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761233.598368:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761233.598374:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761233.598386:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504019200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761233.598393:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761233.598403:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761233.598407:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761233.598410:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761233.598415:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1160 00000100:00080000:1.0:1789761233.598419:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761233.598424:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761233.598430:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761233.598443:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504019328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761233.598453:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761233/real 1789761233] req@ffff9018c66a5880 x1876699504018816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761233.598456:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761233.598468:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761233.598472:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761233.598479:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1160 00000100:00080000:1.0:1789761233.598483:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761233.598487:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504018816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761233.598492:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761233.598503:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761233.598507:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789761233.598517:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504019456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761233.598526:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761233.598534:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504018816:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761233.598543:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504018944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761233.598571:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761233.598585:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761233/real 1789761233] req@ffff9018c66a6a00 x1876699504018944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761233.598607:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504018944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761233.598616:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761233.598619:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761233.598623:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504018944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761233.598650:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504019072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761233.598672:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761233.598685:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761233/real 1789761233] req@ffff9018c66a4a80 x1876699504019072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761233.598699:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504019072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761233.598707:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761233.598710:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761233.598713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504019072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761233.598720:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504019200:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761233.598741:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761233.598753:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761233/real 1789761233] req@ffff9018c66a7480 x1876699504019200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761233.598767:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504019200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761233.598781:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761233.598785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761233.598789:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504019200:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761233.598795:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504019328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761233.598816:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761233.598828:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761233/real 1789761233] req@ffff9018c66a7100 x1876699504019328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761233.598841:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504019328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761233.598849:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761233.598852:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761233.598856:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504019328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761233.598862:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504019456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761233.598883:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761233.598895:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761233/real 1789761233] req@ffff9018c66a4380 x1876699504019456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761233.598914:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504019456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761233.598922:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761233.598926:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761233.598929:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504019456:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761234.238176:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761235.262148:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761236.286161:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761237.310203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761238.334144:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761238.718205:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761238.718229:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761238.718232:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761238.718240:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1165 00000100:00080000:0.0:1789761238.718271:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761238.718280:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761238.718318:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504019584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761238.718341:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761238.718344:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761238.718346:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761238.718349:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1165 00000100:00080000:0.0:1789761238.718352:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761238.718355:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761238.718363:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761238.718373:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504019712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761238.718379:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504019584:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761238.718384:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761238.718387:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761238.718388:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761238.718391:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1165 00000100:00080000:0.0:1789761238.718393:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761238.718395:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761238.718399:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761238.718406:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504019840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761238.718413:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761238.718415:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761238.718416:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761238.718419:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1165 00000100:00080000:0.0:1789761238.718428:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761238.718430:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761238.718433:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789761238.718438:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761238.718442:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504019968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761238.718451:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761238.718454:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761238.718455:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761238.718457:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1165 00000100:00080000:0.0:1789761238.718460:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761238.718462:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761238.718465:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761238.718471:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504020096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761238.718477:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761238.718480:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761238.718481:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761238.718485:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1165 00000100:00100000:1.0:1789761238.718486:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761238/real 1789761238] req@ffff9018c8b99880 x1876699504019584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761238.718490:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761238.718497:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761238.718505:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761238.718513:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504019584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761238.718516:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504020224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761238.718540:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761238.718544:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761238.718548:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504019584:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761238.718567:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504019712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761238.718583:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761238.718592:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761238/real 1789761238] req@ffff9018c8b98e00 x1876699504019712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761238.718601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504019712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761238.718607:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761238.718609:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761238.718611:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504019712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761238.718616:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504019840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761238.718630:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761238.718640:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761238/real 1789761238] req@ffff9018c8b98a80 x1876699504019840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761238.718654:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504019840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761238.718662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761238.718666:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761238.718670:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504019840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761238.718676:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504019968:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761238.718695:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761238.718703:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761238/real 1789761238] req@ffff9018c8b98700 x1876699504019968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761238.718711:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504019968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761238.718717:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761238.718719:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761238.718721:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504019968:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761238.718725:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504020096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761238.718735:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761238.718741:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761238/real 1789761238] req@ffff9018c8b9bb80 x1876699504020096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761238.718752:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504020096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761238.718760:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761238.718763:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761238.718767:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504020096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761238.718773:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504020224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761238.718793:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761238.718805:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761238/real 1789761238] req@ffff9018c8b99f80 x1876699504020224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761238.718813:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504020224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761238.718818:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761238.718820:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761238.718823:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504020224:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761239.358329:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761240.382191:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761241.406214:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761242.430204:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761243.454184:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761243.838142:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761243.838153:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761243.838157:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761243.838168:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1170 00000100:00080000:0.0:1789761243.838179:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761243.838186:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761243.838240:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504020352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761243.838269:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761243.838273:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761243.838276:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761243.838280:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1170 00000100:00080000:0.0:1789761243.838284:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761243.838288:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761243.838298:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761243.838314:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504020480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761243.838319:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504020352:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761243.838337:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761243.838341:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761243.838343:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761243.838349:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1170 00000100:00080000:0.0:1789761243.838354:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761243.838359:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761243.838364:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761243.838380:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761243.838382:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504020608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761243.838396:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761243.838400:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761243.838402:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761243.838407:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1170 00000100:00080000:0.0:1789761243.838411:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761243.838415:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761243.838422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761243.838423:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761243/real 1789761243] req@ffff9018c8b9b480 x1876699504020352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761243.838439:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504020736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761243.838442:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504020352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761243.838472:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761243.838475:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761243.838481:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504020352:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761243.838484:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761243.838489:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761243.838492:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761243.838497:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1170 00000100:00080000:0.0:1789761243.838503:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761243.838503:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504020480:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761243.838521:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789761243.838527:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789761243.838532:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761243.838538:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761243/real 1789761243] req@ffff9018c8b9b800 x1876699504020480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761243.838550:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504020864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761243.838551:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504020480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761243.838580:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761243.838586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761243.838589:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761243.838590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504020480:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761243.838599:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789761243.838599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504020608:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761243.838604:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761243.838612:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1170 00000800:00000100:1.0:1789761243.838617:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761243.838618:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761243.838626:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761243.838629:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761243/real 1789761243] req@ffff9018c8b9aa00 x1876699504020608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789761243.838635:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761243.838640:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504020608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761243.838650:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789761243.838652:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504020992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761243.838657:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761243.838667:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504020608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761243.838678:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504020736:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761243.838703:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761243.838717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761243/real 1789761243] req@ffff9018c8b98000 x1876699504020736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761243.838730:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504020736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761243.838740:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761243.838743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761243.838747:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504020736:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761243.838763:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504020864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761243.838790:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761243.838798:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761243/real 1789761243] req@ffff9018c8b9a680 x1876699504020864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761243.838806:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504020864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761243.838812:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761243.838814:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761243.838816:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504020864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761243.838819:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504020992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761243.838830:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761243.838836:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761243/real 1789761243] req@ffff9018c8b9b100 x1876699504020992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761243.838843:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504020992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761243.838855:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761243.838858:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761243.838859:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504020992:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761244.478228:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761245.502215:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761246.526218:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761247.550160:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761248.574212:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761248.958097:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761248.958105:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761248.958109:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761248.958117:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1175 00000100:00080000:0.0:1789761248.958127:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761248.958134:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761248.958186:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504021120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761248.958210:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761248.958215:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761248.958217:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761248.958222:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1175 00000100:00080000:0.0:1789761248.958226:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761248.958230:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761248.958240:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761248.958254:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504021248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761248.958254:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504021120:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761248.958264:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761248.958267:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761248.958269:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761248.958272:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1175 00000100:00080000:0.0:1789761248.958276:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761248.958284:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761248.958288:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761248.958298:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504021376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761248.958307:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761248.958310:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761248.958312:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761248.958315:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1175 00000100:00080000:0.0:1789761248.958318:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761248.958319:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761248.958322:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761248.958326:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761248.958335:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504021504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761248.958344:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761248.958347:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761248.958349:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761248.958352:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1175 00000100:00080000:0.0:1789761248.958355:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761248.958359:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761248.958363:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761248.958366:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761248/real 1789761248] req@ffff9018c8b9a300 x1876699504021120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761248.958372:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504021632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761248.958382:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761248.958385:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761248.958387:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761248.958387:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504021120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761248.958392:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1175 00000100:00080000:0.0:1789761248.958396:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761248.958398:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761248.958400:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789761248.958401:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:0.0:1789761248.958404:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761248.958406:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504021120:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761248.958412:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504021760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761248.958427:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504021248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761248.958451:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761248.958464:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761248/real 1789761248] req@ffff9018c8b9ad80 x1876699504021248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761248.958473:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504021248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761248.958479:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761248.958481:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761248.958483:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504021248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761248.958488:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504021376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761248.958509:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761248.958515:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761248/real 1789761248] req@ffff9018c8b99500 x1876699504021376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761248.958523:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504021376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761248.958528:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761248.958530:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761248.958538:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504021376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761248.958542:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504021504:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761248.958553:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761248.958560:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761248/real 1789761248] req@ffff9018c8b99c00 x1876699504021504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761248.958568:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504021504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761248.958573:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761248.958575:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761248.958577:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504021504:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761248.958580:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504021632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761248.958590:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761248.958595:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761248/real 1789761248] req@ffff9018c8b98380 x1876699504021632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761248.958608:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504021632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761248.958613:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761248.958615:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761248.958617:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504021632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761248.958620:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504021760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761248.958631:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761248.958637:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761248/real 1789761248] req@ffff9018c8b9b100 x1876699504021760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761248.958643:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504021760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761248.958649:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761248.958650:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761248.958652:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504021760:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761249.598217:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761250.622186:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761251.646169:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761252.670287:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761253.694168:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761254.078164:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761254.078172:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761254.078176:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761254.078184:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1180 00000100:00080000:2.0:1789761254.078195:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761254.078202:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761254.078239:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504021888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761254.078263:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761254.078266:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761254.078267:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761254.078270:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1180 00000100:00080000:2.0:1789761254.078273:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761254.078276:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761254.078285:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761254.078295:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504022016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761254.078306:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761254.078309:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761254.078311:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761254.078315:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1180 00000100:00080000:2.0:1789761254.078319:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761254.078323:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761254.078328:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761254.078337:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504022144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761254.078345:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761254.078349:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504021888:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761254.078349:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761254.078352:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761254.078360:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1180 00000100:00080000:2.0:1789761254.078363:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761254.078365:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761254.078368:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761254.078377:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504022272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761254.078384:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761254.078387:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761254.078388:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761254.078391:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1180 00000100:00080000:2.0:1789761254.078393:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761254.078395:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761254.078398:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761254.078408:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504022400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761254.078416:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761254.078425:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761254.078429:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761254.078432:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761254.078437:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1180 00000100:00080000:2.0:1789761254.078442:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761254.078446:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761254.078452:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761254/real 1789761254] req@ffff9018d0f25c00 x1876699504021888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761254.078453:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761254.078474:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504022528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761254.078475:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504021888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761254.078507:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761254.078511:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761254.078515:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504021888:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761254.078521:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504022016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761254.078535:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761254.078543:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761254/real 1789761254] req@ffff9018d0f26a00 x1876699504022016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761254.078550:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504022016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761254.078556:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761254.078558:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761254.078560:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504022016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761254.078576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504022144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761254.078587:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761254.078593:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761254/real 1789761254] req@ffff9018d0f27100 x1876699504022144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761254.078600:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504022144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761254.078605:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761254.078607:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761254.078609:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504022144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761254.078612:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504022272:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761254.078622:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761254.078627:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761254/real 1789761254] req@ffff9018d0f26300 x1876699504022272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761254.078634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504022272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761254.078639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761254.078640:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761254.078642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504022272:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761254.078645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504022400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761254.078654:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761254.078660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761254/real 1789761254] req@ffff9018d0f25f80 x1876699504022400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761254.078667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504022400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761254.078672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761254.078673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761254.078675:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504022400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761254.078678:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504022528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761254.078687:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761254.078692:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761254/real 1789761254] req@ffff9018d0f26d80 x1876699504022528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761254.078698:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504022528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761254.078703:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761254.078705:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761254.078707:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504022528:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761254.718219:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761255.742194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761256.766201:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761257.790205:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761258.814215:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761259.198141:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761259.198151:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761259.198155:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761259.198164:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1185 00000100:00080000:1.0:1789761259.198175:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761259.198181:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761259.198214:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504022656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761259.198244:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761259.198253:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761259.198254:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761259.198258:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1185 00000100:00080000:1.0:1789761259.198262:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761259.198267:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761259.198276:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761259.198288:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504022656:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761259.198291:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504022784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761259.198304:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761259.198308:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761259.198311:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761259.198315:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1185 00000100:00080000:1.0:1789761259.198319:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761259.198322:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761259.198327:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761259.198339:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504022912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761259.198349:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761259.198352:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761259.198355:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761259.198359:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761259.198364:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1185 00000100:00080000:1.0:1789761259.198368:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761259.198373:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761259.198378:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761259.198399:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504023040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761259.198407:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761259/real 1789761259] req@ffff9018c66a5180 x1876699504022656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761259.198422:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761259.198428:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761259.198431:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761259.198437:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1185 00000100:00100000:0.0:1789761259.198439:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504022656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761259.198441:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761259.198449:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789761259.198461:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1789761259.198462:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761259.198468:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761259.198473:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504022656:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761259.198479:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504023168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761259.198490:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761259.198494:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761259.198497:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761259.198500:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504022784:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789761259.198505:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1185 00000100:00080000:1.0:1789761259.198510:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761259.198514:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761259.198519:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761259.198529:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761259.198534:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504023296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761259.198543:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761259/real 1789761259] req@ffff9018c66a6680 x1876699504022784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761259.198581:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504022784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761259.198592:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761259.198596:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761259.198601:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504022784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761259.198608:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504022912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761259.198630:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761259.198641:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761259/real 1789761259] req@ffff9018c66a4700 x1876699504022912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761259.198650:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504022912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761259.198660:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761259.198663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761259.198667:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504022912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761259.198674:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504023040:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761259.198694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761259.198707:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761259/real 1789761259] req@ffff9018c66a6300 x1876699504023040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761259.198719:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504023040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761259.198725:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761259.198728:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761259.198730:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504023040:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761259.198743:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504023168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761259.198758:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761259.198766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761259/real 1789761259] req@ffff9018c66a4e00 x1876699504023168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761259.198774:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504023168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761259.198779:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761259.198781:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761259.198783:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504023168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761259.198797:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504023296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761259.198808:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761259.198817:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761259/real 1789761259] req@ffff9018c66a5500 x1876699504023296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761259.198829:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504023296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761259.198837:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761259.198841:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761259.198844:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504023296:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761259.838186:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761260.862225:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761261.886193:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761262.910181:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761263.934185:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761264.318157:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761264.318168:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761264.318173:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761264.318183:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1190 00000100:00080000:1.0:1789761264.318197:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761264.318210:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761264.318252:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504023424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761264.318282:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761264.318285:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761264.318286:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761264.318289:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1190 00000100:00080000:1.0:1789761264.318293:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761264.318299:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761264.318310:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761264.318327:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504023424:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761264.318333:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504023552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761264.318353:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761264.318358:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761264.318362:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761264.318370:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1190 00000100:00080000:1.0:1789761264.318376:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761264.318383:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761264.318391:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761264.318397:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761264.318410:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504023680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761264.318420:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761264.318423:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761264.318424:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761264.318428:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1190 00000100:00080000:1.0:1789761264.318431:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761264.318436:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761264.318442:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761264.318450:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761264/real 1789761264] req@ffff9018c66a6d80 x1876699504023424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761264.318453:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504023808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761264.318484:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761264.318494:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761264.318497:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761264.318502:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1190 00000100:00080000:1.0:1789761264.318506:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761264.318509:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761264.318512:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761264.318516:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504023424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761264.318525:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504023936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761264.318540:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761264.318543:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761264.318548:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761264.318549:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789761264.318555:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761264.318561:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1190 00000100:00100000:2.0:1789761264.318561:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504023424:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761264.318570:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761264.318576:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761264.318581:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761264.318591:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504024064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761264.318611:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504023552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761264.318637:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761264.318648:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761264/real 1789761264] req@ffff9018c66a5c00 x1876699504023552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761264.318657:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504023552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761264.318667:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761264.318670:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761264.318672:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504023552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761264.318676:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504023680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761264.318687:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761264.318693:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761264/real 1789761264] req@ffff9018c66a7b80 x1876699504023680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761264.318700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504023680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761264.318705:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761264.318707:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761264.318709:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504023680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761264.318713:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504023808:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761264.318724:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761264.318730:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761264/real 1789761264] req@ffff9018c66a4000 x1876699504023808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761264.318737:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504023808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761264.318742:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761264.318744:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761264.318746:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504023808:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761264.318750:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504023936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761264.318760:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761264.318767:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761264/real 1789761264] req@ffff9018c66a5f80 x1876699504023936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761264.318774:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504023936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761264.318779:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761264.318781:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761264.318782:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504023936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761264.318786:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504024064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761264.318797:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761264.318802:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761264/real 1789761264] req@ffff9018c66a7800 x1876699504024064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761264.318809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504024064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761264.318813:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761264.318815:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761264.318817:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504024064:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761264.958191:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761265.982223:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761267.006180:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761268.030161:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761269.054182:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761269.438109:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761269.438117:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761269.438121:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761269.438129:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1195 00000100:00080000:2.0:1789761269.438137:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761269.438181:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761269.438216:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504024192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761269.438244:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761269.438248:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761269.438249:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761269.438252:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1195 00000100:00080000:2.0:1789761269.438256:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761269.438259:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761269.438268:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761269.438279:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504024320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761269.438284:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504024192:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761269.438291:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761269.438296:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761269.438298:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761269.438303:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1195 00000100:00080000:2.0:1789761269.438307:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761269.438311:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761269.438317:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761269.438343:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504024448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761269.438346:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761269.438363:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761269.438366:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761269.438368:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761269.438372:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1195 00000100:00080000:2.0:1789761269.438376:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761269.438379:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761269.438383:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761269.438386:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761269/real 1789761269] req@ffff9018d0f27800 x1876699504024192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761269.438396:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504024576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761269.438404:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761269.438408:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504024192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761269.438408:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761269.438412:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761269.438418:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1195 00000100:00080000:2.0:1789761269.438423:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761269.438429:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761269.438433:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761269.438434:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789761269.438440:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504024704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761269.438448:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761269.438451:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761269.438452:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761269.438454:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789761269.438457:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1195 00000100:00080000:2.0:1789761269.438461:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761269.438464:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504024192:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761269.438467:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761269.438473:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761269.438480:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504024832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761269.438492:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504024320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761269.438515:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761269.438534:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761269/real 1789761269] req@ffff9018d0f25180 x1876699504024320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761269.438543:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504024320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761269.438549:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761269.438551:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761269.438553:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504024320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761269.438557:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504024448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761269.438570:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761269.438576:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761269/real 1789761269] req@ffff9018d0f24a80 x1876699504024448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761269.438584:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504024448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761269.438589:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761269.438591:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761269.438594:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504024448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761269.438597:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504024576:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761269.438607:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761269.438613:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761269/real 1789761269] req@ffff9018d0f24380 x1876699504024576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761269.438620:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504024576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761269.438626:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761269.438628:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761269.438630:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504024576:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761269.438634:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504024704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761269.438643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761269.438649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761269/real 1789761269] req@ffff9018d0f25500 x1876699504024704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761269.438656:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504024704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761269.438661:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761269.438663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761269.438665:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504024704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761269.438676:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504024832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761269.438687:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761269.438693:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761269/real 1789761269] req@ffff9018d0f24000 x1876699504024832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761269.438701:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504024832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761269.438705:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761269.438707:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761269.438709:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504024832:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761270.078204:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761271.102149:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761272.126191:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761273.150139:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761274.174143:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761274.558174:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761274.558182:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761274.558185:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761274.558193:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1201 00000100:00080000:2.0:1789761274.558202:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761274.558207:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761274.558236:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504024960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761274.558259:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761274.558262:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761274.558263:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761274.558266:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1201 00000100:00080000:2.0:1789761274.558268:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761274.558270:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761274.558278:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761274.558286:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504025088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761274.558292:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761274.558295:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761274.558296:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761274.558299:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1201 00000100:00080000:2.0:1789761274.558301:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761274.558303:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761274.558305:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761274.558311:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504025216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761274.558316:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761274.558319:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761274.558320:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761274.558323:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1201 00000100:00080000:2.0:1789761274.558325:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761274.558362:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504024960:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761274.558385:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761274.558389:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761274.558399:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504025344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761274.558407:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761274.558409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761274.558410:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761274.558413:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1201 00000100:00080000:2.0:1789761274.558415:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761274.558417:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761274.558420:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761274.558426:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504025472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761274.558432:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761274.558434:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761274.558435:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761274.558437:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1201 00000100:00080000:2.0:1789761274.558438:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761274.558440:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761274.558442:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761274.558447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504025600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761274.559561:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761274.559608:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761274/real 1789761274] req@ffff9018d0f24700 x1876699504024960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761274.559629:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504024960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761274.559638:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761274.559641:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761274.559645:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504024960:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761274.559652:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504025088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761274.559673:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761274.559682:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761274/real 1789761274] req@ffff9018d0f26680 x1876699504025088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761274.559690:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504025088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761274.559695:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761274.559697:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761274.559699:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504025088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761274.559703:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504025216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761274.559714:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761274.559720:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761274/real 1789761274] req@ffff9018d0f24e00 x1876699504025216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761274.559727:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504025216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761274.559732:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761274.559733:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761274.559735:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504025216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761274.559754:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504025344:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761274.559765:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761274.559771:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761274/real 1789761274] req@ffff9018d0f27480 x1876699504025344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761274.559779:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504025344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761274.559784:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761274.559785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761274.559787:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504025344:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761274.559790:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504025472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761274.559800:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761274.559806:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761274/real 1789761274] req@ffff9018d0f24000 x1876699504025472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761274.559812:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504025472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761274.559818:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761274.559820:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761274.559822:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504025472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761274.559825:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504025600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761274.559834:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761274.559839:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761274/real 1789761274] req@ffff9018d0f25500 x1876699504025600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761274.559846:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504025600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761274.559851:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761274.559853:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761274.559855:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504025600:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761275.198142:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761276.222197:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761277.246166:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761278.270153:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761279.294186:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761279.678151:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761279.678183:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761279.678186:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761279.678196:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1206 00000100:00080000:1.0:1789761279.678207:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761279.678221:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761279.678259:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504025728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761279.678282:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761279.678286:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761279.678288:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761279.678298:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1206 00000100:00080000:1.0:1789761279.678303:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761279.678307:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761279.678318:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761279.678324:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504025728:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761279.678335:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504025856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761279.678356:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761279.678360:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761279.678362:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761279.678367:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1206 00000100:00080000:1.0:1789761279.678374:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761279.678378:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761279.678383:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761279.678391:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761279.678397:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504025984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761279.678409:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761279.678412:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761279.678414:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761279.678418:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1206 00000100:00080000:1.0:1789761279.678422:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761279.678426:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761279.678431:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761279.678438:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761279/real 1789761279] req@ffff9018c66a7800 x1876699504025728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761279.678445:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504026112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761279.678468:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761279.678474:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789761279.678475:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504025728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761279.678477:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761279.678486:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1206 00000100:00080000:0.0:1789761279.678491:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761279.678492:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761279.678496:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789761279.678502:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761279.678503:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504025728:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789761279.678513:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761279.678529:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504026240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761279.678532:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504025856:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789761279.678549:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761279.678553:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761279.678556:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761279.678562:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1206 00000800:00000100:0.0:1789761279.678566:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761279.678569:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761279.678577:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761279.678583:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761279.678585:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761279/real 1789761279] req@ffff9018c66a5f80 x1876699504025856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761279.678603:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504025856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761279.678604:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504026368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761279.678629:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761279.678646:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761279.678649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504025856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761279.678655:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504025984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761279.678669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761279.678677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761279/real 1789761279] req@ffff9018c66a4000 x1876699504025984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761279.678685:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504025984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761279.678705:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761279.678707:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761279.678710:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504025984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761279.678714:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504026112:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761279.678727:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761279.678735:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761279/real 1789761279] req@ffff9018c66a7b80 x1876699504026112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761279.678743:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504026112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761279.678756:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761279.678758:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761279.678761:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504026112:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761279.678785:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504026240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761279.678798:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761279.678806:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761279/real 1789761279] req@ffff9018c66a5c00 x1876699504026240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761279.678814:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504026240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761279.678820:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761279.678822:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761279.678825:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504026240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761279.678828:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504026368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761279.678838:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761279.678843:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761279/real 1789761279] req@ffff9018c66a6d80 x1876699504026368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761279.678850:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504026368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761279.678862:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761279.678865:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761279.678867:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504026368:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761280.318171:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761281.342179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761282.366155:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761283.390188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761284.414184:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761284.798142:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761284.798152:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761284.798156:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761284.798165:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1211 00000100:00080000:2.0:1789761284.798177:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761284.798185:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761284.798224:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504026496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761284.798265:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761284.798269:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761284.798272:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761284.798277:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1211 00000100:00080000:2.0:1789761284.798281:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761284.798285:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761284.798294:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761284.798305:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504026624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761284.798314:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761284.798318:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789761284.798320:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504026496:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761284.798321:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761284.798336:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1211 00000100:00080000:2.0:1789761284.798342:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761284.798345:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761284.798350:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761284.798360:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504026752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761284.798367:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761284.798370:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761284.798371:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761284.798374:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1211 00000800:00000100:0.0:1789761284.798379:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761284.798379:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761284.798385:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761284.798390:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761284.798399:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504026880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761284.798412:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761284.798414:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761284.798416:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761284.798418:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1211 00000100:00080000:2.0:1789761284.798421:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761284.798424:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761284/real 1789761284] req@ffff9018d0f24380 x1876699504026496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761284.798430:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761284.798441:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761284.798448:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504026496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761284.798457:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504027008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761284.798459:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761284.798467:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761284.798473:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504026496:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761284.798473:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761284.798481:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761284.798484:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761284.798487:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1211 00000100:00080000:2.0:1789761284.798491:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761284.798493:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761284.798497:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761284.798502:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504026624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761284.798508:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504027136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761284.798533:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761284.798549:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761284/real 1789761284] req@ffff9018d0f24a80 x1876699504026624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761284.798562:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504026624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761284.798572:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761284.798576:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761284.798580:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504026624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761284.798586:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504026752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761284.798602:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761284.798611:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761284/real 1789761284] req@ffff9018d0f25180 x1876699504026752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761284.798619:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504026752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761284.798624:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761284.798626:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761284.798628:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504026752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761284.798632:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504026880:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761284.798643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761284.798649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761284/real 1789761284] req@ffff9018d0f27800 x1876699504026880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761284.798656:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504026880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761284.798662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761284.798663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761284.798665:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504026880:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761284.798669:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504027008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761284.798679:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761284.798685:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761284/real 1789761284] req@ffff9018d0f26d80 x1876699504027008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761284.798693:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504027008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761284.798698:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761284.798700:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761284.798702:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504027008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761284.798714:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504027136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761284.798724:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761284.798730:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761284/real 1789761284] req@ffff9018d0f25f80 x1876699504027136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761284.798737:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504027136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761284.798742:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761284.798743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761284.798745:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504027136:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761285.438206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761286.462161:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761287.486213:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761288.510160:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761289.534202:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761289.918235:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761289.918248:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761289.918255:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761289.918270:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1216 00000100:00080000:1.0:1789761289.918285:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761289.918296:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761289.918347:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504027264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761289.918379:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761289.918384:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761289.918385:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761289.918390:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1216 00000100:00080000:1.0:1789761289.918393:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761289.918397:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761289.918408:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761289.918420:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504027392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761289.918429:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761289.918435:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789761289.918437:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504027264:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761289.918438:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761289.918448:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1216 00000100:00080000:1.0:1789761289.918451:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761289.918454:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761289.918457:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761289.918476:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504027520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761289.918484:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761289.918487:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761289.918489:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761289.918491:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1216 00000100:00080000:1.0:1789761289.918505:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761289.918509:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761289.918514:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789761289.918523:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761289.918526:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504027648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761289.918536:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761289.918540:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761289.918542:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761289.918546:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1216 00000100:00080000:1.0:1789761289.918551:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761289.918557:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761289.918563:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761289.918579:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761289/real 1789761289] req@ffff9018c66a5500 x1876699504027264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761289.918582:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504027776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761289.918612:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761289.918617:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504027264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761289.918619:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761289.918625:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761289.918635:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1216 00000100:00080000:1.0:1789761289.918641:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761289.918649:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789761289.918651:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1789761289.918657:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761289.918658:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761289.918667:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504027264:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761289.918680:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504027904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761289.918697:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504027392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761289.918739:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761289.918755:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761289/real 1789761289] req@ffff9018c66a4e00 x1876699504027392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761289.918768:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504027392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761289.918776:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761289.918779:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761289.918782:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504027392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761289.918787:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504027520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761289.918803:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761289.918811:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761289/real 1789761289] req@ffff9018c66a6300 x1876699504027520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761289.918835:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504027520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761289.918842:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761289.918845:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761289.918848:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504027520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761289.918854:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504027648:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761289.918911:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761289.918921:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761289/real 1789761289] req@ffff9018c66a4700 x1876699504027648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761289.918931:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504027648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761289.918938:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761289.918940:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761289.918943:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504027648:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761289.918947:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504027776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761289.918981:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761289.919009:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761289/real 1789761289] req@ffff9018c66a6680 x1876699504027776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761289.919043:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504027776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761289.919098:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761289.919101:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761289.919104:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504027776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761289.919122:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504027904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761289.919142:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761289.919152:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761289/real 1789761289] req@ffff9018c66a5180 x1876699504027904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761289.919167:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504027904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761289.919176:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761289.919180:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761289.919184:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504027904:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761290.558146:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761291.582242:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761292.606144:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761293.630190:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761294.654227:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761295.038198:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761295.038207:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761295.038211:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761295.038219:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1221 00000100:00080000:2.0:1789761295.038228:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761295.038234:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761295.038269:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504028032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761295.038292:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761295.038295:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761295.038296:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761295.038299:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1221 00000100:00080000:2.0:1789761295.038301:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761295.038304:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761295.038311:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761295.038322:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504028160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761295.038330:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761295.038332:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761295.038334:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761295.038336:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1221 00000100:00080000:2.0:1789761295.038338:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761295.038341:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761295.038344:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761295.038351:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504028032:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761295.038356:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504028288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761295.038375:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761295.038379:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761295.038382:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761295.038387:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1221 00000100:00080000:2.0:1789761295.038391:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761295.038396:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761295.038402:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761295.038417:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504028416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761295.038428:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761295.038432:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761295.038434:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761295.038439:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1221 00000100:00080000:2.0:1789761295.038444:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761295.038449:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761295.038455:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761295.038472:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504028544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761295.038478:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504028160:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761295.038503:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761295.038508:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761295.038511:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761295.038520:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1221 00000100:00080000:2.0:1789761295.038526:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761295.038529:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761295.038536:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504028288:192.168.203.158@tcp:38:kworker.0 00000008:00100000:2.0:1789761295.038537:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761295.038559:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504028672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761295.038561:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504028416:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761295.038588:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504028544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761295.038617:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504028672:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761295.678289:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761296.702154:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761297.726247:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761298.110315:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:3.0:1789761298.110373:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761298.110395:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761298.110441:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761298.110453:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761298.110465:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761298.110476:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789761298.110485:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761295/real 1789761298] req@ffff9018d0f26a00 x1876699504028288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1789761298.110487:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789761298.110521:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504028288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761298.110536:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761298.110542:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761298.110546:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504028288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761298.110557:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761295/real 1789761298] req@ffff9018d0f25c00 x1876699504028416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761298.110573:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504028416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761298.110582:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761298.110587:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761298.110591:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504028416:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761298.110604:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761295/real 1789761298] req@ffff9018d0f25f80 x1876699504028544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761298.110617:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504028544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761298.110625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761298.110629:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761298.110632:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504028544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761298.110639:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761295/real 1789761298] req@ffff9018d0f26300 x1876699504028032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761298.110651:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504028032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761298.110659:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761298.110663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761298.110666:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504028032:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761298.110675:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761295/real 1789761298] req@ffff9018d0f27100 x1876699504028160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761298.110686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504028160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761298.110694:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761298.110697:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761298.110701:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504028160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761298.110738:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761295/real 1789761298] req@ffff9018d0f26d80 x1876699504028672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761298.110752:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504028672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761298.110761:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761298.110765:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761298.110769:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504028672:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761298.750291:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761299.774241:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761300.158223:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761300.158234:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761300.158239:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761300.158253:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1226 00000100:00080000:3.0:1789761300.158278:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761300.158293:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761300.158340:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504028800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761300.158373:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761300.158378:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761300.158381:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761300.158387:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1226 00000100:00080000:3.0:1789761300.158392:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761300.158398:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761300.158411:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761300.158422:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504028800:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761300.158433:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504028928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761300.158447:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761300.158451:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761300.158453:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761300.158458:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1226 00000100:00080000:3.0:1789761300.158463:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761300.158468:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761300.158474:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761300.158487:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504029056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761300.158499:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761300.158503:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761300.158505:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761300.158509:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1226 00000100:00080000:3.0:1789761300.158514:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761300.158515:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761300.158523:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761300.158531:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761300.158548:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504029184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761300.158560:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761300.158564:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761300.158567:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761300.158574:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1226 00000100:00100000:0.0:1789761300.158575:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761300/real 1789761300] req@ffff9018d070df80 x1876699504028800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761300.158581:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761300.158589:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761300.158598:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761300.158609:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504028800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761300.158615:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504029312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761300.158634:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761300.158636:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761300.158643:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761300.158646:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789761300.158648:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761300.158655:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504028800:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761300.158657:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1226 00000100:00080000:3.0:1789761300.158667:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761300.158675:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761300.158682:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761300.158695:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504029440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761300.158704:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504028928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761300.158743:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761300.158777:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761300/real 1789761300] req@ffff9018d070ed80 x1876699504028928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761300.158810:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504028928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761300.158820:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761300.158824:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761300.158828:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504028928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761300.158836:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504029056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761300.158860:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761300.158874:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761300/real 1789761300] req@ffff9018d070f480 x1876699504029056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761300.158888:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504029056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761300.158896:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761300.158906:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761300.158910:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504029056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761300.158917:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504029184:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761300.158943:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761300.158956:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761300/real 1789761300] req@ffff9018d070ca80 x1876699504029184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761300.158970:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504029184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761300.158978:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761300.158982:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761300.158987:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504029184:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761300.158993:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504029312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761300.159014:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761300.159027:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761300/real 1789761300] req@ffff9018d070ce00 x1876699504029312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761300.159089:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504029312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761300.159098:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761300.159102:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761300.159105:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504029312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761300.159126:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504029440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761300.159150:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761300.159164:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761300/real 1789761300] req@ffff9018d070d880 x1876699504029440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761300.159184:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504029440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761300.159193:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761300.159197:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761300.159200:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504029440:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761300.798267:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761301.822232:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761302.846255:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761303.870213:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761304.894192:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761305.278105:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761305.278113:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761305.278118:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761305.278126:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1231 00000100:00080000:3.0:1789761305.278135:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761305.278141:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761305.278171:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504029568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761305.278192:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761305.278195:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761305.278197:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761305.278203:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1231 00000100:00080000:3.0:1789761305.278205:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761305.278208:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761305.278215:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761305.278224:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504029696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761305.278232:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761305.278234:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761305.278235:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761305.278241:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504029568:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761305.278242:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1231 00000100:00080000:3.0:1789761305.278247:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761305.278251:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761305.278255:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761305.278265:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504029824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761305.278275:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761305.278278:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761305.278281:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761305.278285:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1231 00000100:00080000:3.0:1789761305.278293:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761305.278297:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789761305.278301:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789761305.278312:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761305.278328:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504029952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761305.278336:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761305.278339:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761305/real 1789761305] req@ffff9018d070fb80 x1876699504029568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761305.278339:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761305.278342:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761305.278349:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1231 00000100:00080000:3.0:1789761305.278353:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761305.278357:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761305.278364:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761305.278374:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504029568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761305.278374:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504030080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761305.278406:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761305.278406:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761305.278413:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761305.278417:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504029568:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761305.278419:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761305.278423:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761305.278429:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1231 00000100:00080000:3.0:1789761305.278433:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761305.278437:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761305.278443:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504029696:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789761305.278444:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761305.278457:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504030208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761305.278477:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761305.278495:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761305/real 1789761305] req@ffff9018d070c000 x1876699504029696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761305.278510:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504029696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761305.278526:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761305.278530:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761305.278534:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504029696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761305.278540:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504029824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761305.278562:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761305.278574:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761305/real 1789761305] req@ffff9018d070dc00 x1876699504029824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761305.278587:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504029824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761305.278594:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761305.278596:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761305.278598:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504029824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761305.278602:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504029952:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761305.278618:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761305.278625:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761305/real 1789761305] req@ffff9018d070e300 x1876699504029952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761305.278633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504029952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761305.278638:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761305.278640:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761305.278642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504029952:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761305.278646:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504030080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761305.278656:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761305.278662:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761305/real 1789761305] req@ffff9018d070d180 x1876699504030080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761305.278668:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504030080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761305.278673:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761305.278675:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761305.278677:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504030080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761305.278689:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504030208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761305.278699:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761305.278705:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761305/real 1789761305] req@ffff9018d070f800 x1876699504030208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761305.278712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504030208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761305.278717:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761305.278718:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761305.278720:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504030208:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761305.918207:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761306.942191:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761307.966195:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761308.990221:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761310.014159:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761310.398133:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761310.398144:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761310.398148:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761310.398158:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1236 00000100:00080000:3.0:1789761310.398169:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761310.398177:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761310.398211:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504030336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761310.398233:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761310.398236:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761310.398237:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761310.398240:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1236 00000100:00080000:3.0:1789761310.398243:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761310.398245:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761310.398253:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761310.398262:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504030464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761310.398269:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761310.398275:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789761310.398276:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504030336:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761310.398277:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761310.398285:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1236 00000100:00080000:3.0:1789761310.398288:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761310.398290:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761310.398293:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761310.398302:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504030592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761310.398308:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761310.398310:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761310.398312:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761310.398315:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1236 00000100:00080000:3.0:1789761310.398318:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761310.398320:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761310.398323:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761310.398332:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504030720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761310.398338:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761310.398339:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761310.398343:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761310.398346:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761310.398349:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1236 00000100:00080000:3.0:1789761310.398359:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761310.398362:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761310.398367:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761310.398374:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504030848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761310.398380:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761310.398382:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761310.398384:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761310.398386:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1236 00000100:00080000:3.0:1789761310.398387:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761310.398390:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761310.398393:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761310.398403:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761310/real 1789761310] req@ffff9018d070d500 x1876699504030336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761310.398403:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504030976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761310.398440:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504030336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761310.398449:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761310.398452:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761310.398455:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504030336:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761310.398473:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504030464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761310.398492:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761310.398501:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761310/real 1789761310] req@ffff9018d070e680 x1876699504030464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761310.398509:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504030464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761310.398515:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761310.398517:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761310.398519:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504030464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761310.398523:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504030592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761310.398534:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761310.398539:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761310/real 1789761310] req@ffff9018d070f100 x1876699504030592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761310.398546:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504030592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761310.398556:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761310.398558:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761310.398560:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504030592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761310.398564:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504030720:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761310.398574:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761310.398580:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761310/real 1789761310] req@ffff9018d070f800 x1876699504030720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761310.398587:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504030720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761310.398592:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761310.398594:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761310.398596:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504030720:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761310.398600:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504030848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761310.398609:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761310.398614:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761310/real 1789761310] req@ffff9018d070d180 x1876699504030848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761310.398621:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504030848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761310.398626:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761310.398628:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761310.398630:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504030848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761310.398634:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504030976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761310.398644:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761310.398649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761310/real 1789761310] req@ffff9018d070e300 x1876699504030976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761310.398656:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504030976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761310.398668:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761310.398670:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761310.398672:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504030976:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761311.038209:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761312.062146:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761313.086213:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761314.110167:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761315.134191:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761315.518121:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761315.518135:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761315.518140:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761315.518153:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1242 00000100:00080000:3.0:1789761315.518167:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761315.518176:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761315.518228:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504031104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761315.518266:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761315.518270:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761315.518273:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761315.518279:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1242 00000100:00080000:3.0:1789761315.518285:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761315.518291:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761315.518305:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761315.518313:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504031104:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761315.518329:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504031232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761315.518346:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761315.518350:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761315.518352:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761315.518355:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1242 00000100:00080000:3.0:1789761315.518359:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761315.518362:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761315.518367:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761315.518379:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504031360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761315.518386:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761315.518389:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761315.518391:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761315.518394:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1242 00000100:00080000:3.0:1789761315.518396:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761315.518398:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761315.518404:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789761315.518405:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761315.518418:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504031488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761315.518434:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761315.518437:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761315.518439:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761315.518442:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1242 00000100:00080000:3.0:1789761315.518445:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761315.518449:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761315.518456:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761315.518463:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761315/real 1789761315] req@ffff9018d070dc00 x1876699504031104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761315.518468:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504031616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761315.518477:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761315.518496:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761315.518502:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761315.518516:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1242 00000100:00080000:3.0:1789761315.518521:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761315.518525:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761315.518528:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761315.518530:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504031104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761315.518543:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504031744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761315.518547:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761315.518565:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761315.518576:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504031104:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761315.518609:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504031232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761315.518641:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761315.518653:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761315/real 1789761315] req@ffff9018d070c000 x1876699504031232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761315.518663:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504031232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761315.518669:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761315.518685:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761315.518688:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504031232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761315.518693:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504031360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761315.518711:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761315.518723:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761315/real 1789761315] req@ffff9018d070fb80 x1876699504031360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761315.518736:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504031360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761315.518745:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761315.518749:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761315.518753:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504031360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761315.518760:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504031488:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761315.518779:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761315.518787:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761315/real 1789761315] req@ffff9018d070d880 x1876699504031488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761315.518802:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504031488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761315.518810:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761315.518814:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761315.518817:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504031488:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761315.518824:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504031616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761315.518846:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761315.518856:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761315/real 1789761315] req@ffff9018d070ce00 x1876699504031616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761315.518865:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504031616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761315.518870:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761315.518872:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761315.518874:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504031616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761315.518879:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504031744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761315.518892:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761315.518898:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761315/real 1789761315] req@ffff9018d070ca80 x1876699504031744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761315.518906:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504031744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761315.518915:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761315.518917:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761315.518919:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504031744:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761316.158207:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761317.182186:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761318.206212:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761319.230146:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761320.254190:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761320.638154:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761320.638164:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761320.638169:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761320.638180:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1247 00000100:00080000:1.0:1789761320.638193:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761320.638201:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761320.638251:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504031872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761320.638277:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761320.638280:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761320.638281:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761320.638285:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1247 00000100:00080000:1.0:1789761320.638288:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761320.638291:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761320.638302:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761320.638314:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504032000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761320.638324:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761320.638326:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504031872:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761320.638330:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761320.638336:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761320.638342:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1247 00000100:00080000:1.0:1789761320.638346:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761320.638350:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761320.638356:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761320.638373:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504032128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761320.638390:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761320.638393:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761320.638395:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761320.638399:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1247 00000100:00080000:1.0:1789761320.638402:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761320.638405:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761320.638410:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789761320.638414:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761320.638424:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504032256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761320.638432:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761320.638435:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761320.638437:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761320.638439:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1247 00000100:00080000:1.0:1789761320.638441:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761320.638444:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761320.638447:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761320.638454:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504032384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761320.638459:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761320.638462:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761320.638463:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761320.638466:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1247 00000100:00080000:1.0:1789761320.638469:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761320.638472:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761320.638476:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761320.638477:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761320/real 1789761320] req@ffff9018c66a4380 x1876699504031872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761320.638487:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504032512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761320.638498:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504031872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761320.638509:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761320.638513:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761320.638518:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504031872:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761320.638541:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504032000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761320.638577:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761320.638586:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761320/real 1789761320] req@ffff9018c66a7100 x1876699504032000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761320.638595:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504032000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761320.638601:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761320.638605:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761320.638607:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504032000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761320.638611:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504032128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761320.638623:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761320.638629:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761320/real 1789761320] req@ffff9018c66a7480 x1876699504032128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761320.638653:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504032128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761320.638660:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761320.638662:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761320.638680:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504032128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761320.638685:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504032256:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761320.638701:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761320.638708:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761320/real 1789761320] req@ffff9018c66a4a80 x1876699504032256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761320.638716:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504032256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761320.638721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761320.638723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761320.638725:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504032256:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761320.638729:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504032384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761320.638740:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761320.638746:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761320/real 1789761320] req@ffff9018c66a6a00 x1876699504032384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761320.638753:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504032384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761320.638758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761320.638760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761320.638762:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504032384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761320.638765:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504032512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761320.638775:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761320.638780:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761320/real 1789761320] req@ffff9018c66a5880 x1876699504032512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761320.638787:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504032512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761320.638793:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761320.638794:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761320.638796:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504032512:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761321.278183:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761322.302194:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761323.326181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761324.350238:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761325.374186:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761325.758183:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761325.758195:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761325.758200:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761325.758213:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1252 00000100:00080000:2.0:1789761325.758227:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761325.758237:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761325.758289:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504032640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761325.758322:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761325.758325:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761325.758327:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761325.758331:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1252 00000100:00080000:2.0:1789761325.758334:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761325.758338:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761325.758349:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761325.758360:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504032768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761325.758368:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761325.758373:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504032640:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761325.758373:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761325.758375:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761325.758382:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1252 00000100:00080000:2.0:1789761325.758387:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761325.758389:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761325.758393:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761325.758400:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504032896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761325.758407:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761325.758410:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761325.758411:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761325.758414:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1252 00000100:00080000:2.0:1789761325.758415:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761325.758418:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761325.758421:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761325.758430:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504033024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761325.758436:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761325.758439:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761325.758441:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761325.758443:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1252 00000100:00080000:2.0:1789761325.758452:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761325.758455:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761325.758460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761325.758467:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761325.758471:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504033152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761325.758479:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761325.758482:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761325.758484:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761325.758486:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1252 00000100:00080000:2.0:1789761325.758489:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761325.758491:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761325.758494:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761325.758501:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504033280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761325.758535:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761325/real 1789761325] req@ffff9018d0f27800 x1876699504032640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761325.758563:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504032640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761325.758577:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761325.758582:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761325.758587:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504032640:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761325.758628:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504032768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761325.758653:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761325.758667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761325/real 1789761325] req@ffff9018d0f25180 x1876699504032768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761325.758681:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504032768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761325.758690:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761325.758694:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761325.758697:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504032768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761325.758704:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504032896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761325.758725:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761325.758736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761325/real 1789761325] req@ffff9018d0f24a80 x1876699504032896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761325.758751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504032896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761325.758767:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761325.758770:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761325.758774:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504032896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761325.758781:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504033024:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761325.758802:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761325.758813:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761325/real 1789761325] req@ffff9018d0f24380 x1876699504033024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761325.758825:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504033024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761325.758834:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761325.758838:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761325.758841:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504033024:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761325.758847:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504033152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761325.758863:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761325.758872:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761325/real 1789761325] req@ffff9018d0f25500 x1876699504033152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761325.758884:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504033152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761325.758893:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761325.758896:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761325.758901:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504033152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761325.758907:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504033280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761325.758928:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761325.758940:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761325/real 1789761325] req@ffff9018d0f24000 x1876699504033280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761325.758953:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504033280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761325.758970:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761325.758973:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761325.758977:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504033280:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761326.398167:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761327.422180:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761328.446209:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761329.470238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761330.494230:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761330.878146:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761330.878161:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761330.878166:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761330.878180:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1257 00000100:00080000:2.0:1789761330.878205:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761330.878214:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761330.878257:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504033408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761330.878283:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761330.878286:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761330.878288:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761330.878292:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1257 00000100:00080000:2.0:1789761330.878295:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761330.878298:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761330.878308:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761330.878318:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504033536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761330.878327:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761330.878328:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504033408:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761330.878330:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761330.878334:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761330.878339:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1257 00000100:00080000:2.0:1789761330.878341:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761330.878343:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761330.878346:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761330.878353:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504033664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761330.878360:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761330.878362:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761330.878364:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761330.878366:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1257 00000100:00080000:2.0:1789761330.878368:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761330.878371:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761330.878375:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761330.878383:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504033792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761330.878390:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761330.878392:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761330.878394:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761330.878396:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1257 00000100:00080000:2.0:1789761330.878403:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761330.878405:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761330.878408:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761330.878418:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504033920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761330.878420:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761330.878430:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761330.878433:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761330.878434:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761330.878437:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1257 00000100:00080000:2.0:1789761330.878439:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761330.878441:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761330.878444:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761330.878451:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504034048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761330.878484:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761330/real 1789761330] req@ffff9018d0f27480 x1876699504033408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761330.878501:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504033408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761330.878517:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761330.878521:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761330.878526:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504033408:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761330.878553:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504033536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761330.878587:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761330.878597:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761330/real 1789761330] req@ffff9018d0f24e00 x1876699504033536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761330.878606:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504033536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761330.878612:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761330.878615:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761330.878617:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504033536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761330.878622:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504033664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761330.878634:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761330.878640:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761330/real 1789761330] req@ffff9018d0f26680 x1876699504033664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761330.878648:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504033664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761330.878653:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761330.878655:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761330.878657:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504033664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761330.878660:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504033792:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761330.878671:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761330.878677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761330/real 1789761330] req@ffff9018d0f24700 x1876699504033792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761330.878690:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504033792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761330.878695:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761330.878697:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761330.878699:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504033792:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761330.878702:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504033920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761330.878712:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761330.878719:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761330/real 1789761330] req@ffff9018d0f24000 x1876699504033920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761330.878726:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504033920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761330.878731:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761330.878733:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761330.878735:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504033920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761330.878738:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504034048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761330.878748:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761330.878753:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761330/real 1789761330] req@ffff9018d0f25500 x1876699504034048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761330.878761:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504034048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761330.878776:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761330.878780:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761330.878783:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504034048:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761331.518194:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761332.542179:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761333.566200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761334.590210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761335.614159:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761335.998150:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761335.998162:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761335.998167:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761335.998178:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1262 00000100:00080000:2.0:1789761335.998191:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761335.998199:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761335.998245:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504034176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761335.998276:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761335.998280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761335.998281:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761335.998289:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1262 00000100:00080000:2.0:1789761335.998292:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761335.998297:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761335.998309:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761335.998325:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504034304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761335.998329:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504034176:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761335.998334:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761335.998342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761335.998348:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761335.998352:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1262 00000100:00080000:2.0:1789761335.998355:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761335.998358:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761335.998362:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761335.998373:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504034432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761335.998384:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761335.998387:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761335.998389:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761335.998394:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1262 00000100:00080000:2.0:1789761335.998397:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761335.998399:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761335.998403:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761335.998410:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504034560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761335.998420:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761335.998421:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761335.998430:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761335.998433:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761335.998438:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1262 00000100:00080000:2.0:1789761335.998443:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761335.998446:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761335.998450:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761335.998457:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504034688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761335.998465:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761335.998468:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761335.998470:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761335.998476:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1262 00000100:00100000:0.0:1789761335.998478:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761335/real 1789761335] req@ffff9018d0f24380 x1876699504034176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761335.998485:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761335.998492:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761335.998500:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761335.998513:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504034816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761335.998548:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504034176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761335.998559:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761335.998563:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761335.998569:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504034176:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761335.998595:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504034304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761335.998623:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761335.998634:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761335/real 1789761335] req@ffff9018d0f24a80 x1876699504034304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761335.998643:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504034304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761335.998649:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761335.998652:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761335.998654:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504034304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761335.998659:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504034432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761335.998674:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761335.998680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761335/real 1789761335] req@ffff9018d0f25180 x1876699504034432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761335.998689:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504034432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761335.998694:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761335.998696:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761335.998699:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504034432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761335.998703:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504034560:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761335.998715:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761335.998722:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761335/real 1789761335] req@ffff9018d0f27800 x1876699504034560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761335.998729:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504034560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761335.998735:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761335.998737:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761335.998739:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504034560:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761335.998743:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504034688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761335.998755:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761335.998764:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761335/real 1789761335] req@ffff9018d0f26d80 x1876699504034688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761335.998777:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504034688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761335.998786:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761335.998791:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761335.998804:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504034688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761335.998815:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504034816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761335.998854:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761335.998883:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761335/real 1789761335] req@ffff9018d0f27100 x1876699504034816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761335.998893:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504034816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761335.998900:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761335.998903:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761335.998905:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504034816:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761336.638168:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761337.662206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761338.686183:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761339.710233:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761340.734190:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761341.118117:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761341.118125:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761341.118128:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761341.118136:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1267 00000100:00080000:2.0:1789761341.118145:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761341.118152:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761341.118182:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504034944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761341.118201:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761341.118204:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761341.118205:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761341.118208:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1267 00000100:00080000:2.0:1789761341.118210:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761341.118213:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761341.118220:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761341.118228:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504035072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761341.118241:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504034944:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761341.118242:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761341.118251:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761341.118252:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761341.118256:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1267 00000100:00080000:2.0:1789761341.118257:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761341.118259:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761341.118263:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761341.118271:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504035200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761341.118277:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761341.118280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761341.118281:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761341.118284:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1267 00000100:00080000:2.0:1789761341.118286:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761341.118288:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761341.118292:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761341.118299:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504035328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761341.118303:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761341.118318:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761341.118321:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761341.118323:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761341.118326:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1267 00000100:00080000:2.0:1789761341.118330:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761341.118339:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761341.118342:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761341/real 1789761341] req@ffff9018d0f26300 x1876699504034944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761341.118346:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761341.118362:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504034944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761341.118368:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504035456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761341.118372:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761341.118381:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761341.118388:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504034944:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761341.118389:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761341.118396:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761341.118406:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761341.118414:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504035072:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761341.118414:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1267 00000100:00080000:2.0:1789761341.118422:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761341.118428:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761341.118433:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761341.118438:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761341.118446:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504035584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761341.118451:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761341/real 1789761341] req@ffff9018d0f25f80 x1876699504035072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761341.118470:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504035072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761341.118477:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761341.118496:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761341.118499:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504035072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761341.118504:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504035200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761341.118521:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761341.118533:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761341/real 1789761341] req@ffff9018d0f25c00 x1876699504035200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761341.118547:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504035200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761341.118564:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761341.118568:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761341.118572:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504035200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761341.118578:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504035328:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761341.118594:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761341.118602:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761341/real 1789761341] req@ffff9018d0f26a00 x1876699504035328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761341.118610:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504035328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761341.118616:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761341.118618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761341.118620:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504035328:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761341.118624:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504035456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761341.118634:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761341.118640:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761341/real 1789761341] req@ffff9018d0f27100 x1876699504035456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761341.118647:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504035456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761341.118652:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761341.118654:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761341.118656:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504035456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761341.118668:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504035584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761341.118678:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761341.118684:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761341/real 1789761341] req@ffff9018d0f26d80 x1876699504035584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761341.118692:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504035584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761341.118696:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761341.118698:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761341.118700:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504035584:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761341.758178:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761342.782202:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761343.806191:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761344.830152:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761345.854167:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761346.238144:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761346.238153:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761346.238156:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761346.238164:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1272 00000100:00080000:3.0:1789761346.238173:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761346.238179:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761346.238210:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504035712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761346.238231:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761346.238234:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761346.238235:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761346.238238:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1272 00000100:00080000:3.0:1789761346.238241:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761346.238244:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761346.238252:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761346.238261:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504035840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761346.238270:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761346.238271:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504035712:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761346.238281:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761346.238283:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761346.238288:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1272 00000100:00080000:3.0:1789761346.238293:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761346.238297:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761346.238303:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761346.238316:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504035968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761346.238325:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761346.238329:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1789761346.238330:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761346.238332:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761346.238337:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1272 00000100:00080000:3.0:1789761346.238341:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761346.238345:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761346.238349:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761346.238359:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504036096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761346.238367:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761346.238369:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761346.238372:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761346.238373:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761346/real 1789761346] req@ffff9018d070f480 x1876699504035712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761346.238377:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1272 00000100:00080000:3.0:1789761346.238383:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761346.238390:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761346.238395:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761346.238399:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504035712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761346.238406:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504036224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761346.238411:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761346.238416:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761346.238419:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789761346.238421:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761346.238423:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761346.238426:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504035712:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761346.238429:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1272 00000100:00080000:3.0:1789761346.238432:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761346.238434:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761346.238454:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761346.238465:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504036352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761346.238474:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504035840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761346.238501:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761346.238512:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761346/real 1789761346] req@ffff9018d070ed80 x1876699504035840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761346.238521:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504035840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761346.238526:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761346.238529:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761346.238531:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504035840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761346.238535:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504035968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761346.238547:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761346.238553:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761346/real 1789761346] req@ffff9018d070df80 x1876699504035968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761346.238560:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504035968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761346.238565:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761346.238567:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761346.238569:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504035968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761346.238573:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504036096:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761346.238584:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761346.238589:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761346/real 1789761346] req@ffff9018d070ca80 x1876699504036096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761346.238601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504036096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761346.238606:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761346.238608:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761346.238610:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504036096:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761346.238613:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504036224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761346.238623:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761346.238629:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761346/real 1789761346] req@ffff9018d070ce00 x1876699504036224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761346.238636:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504036224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761346.238640:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761346.238642:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761346.238644:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504036224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761346.238656:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504036352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761346.238666:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761346.238672:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761346/real 1789761346] req@ffff9018d070d880 x1876699504036352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761346.238679:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504036352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761346.238688:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761346.238690:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761346.238692:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504036352:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761346.878246:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761347.902231:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761348.926203:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761349.950218:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761350.974206:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761351.358148:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761351.358156:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761351.358160:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761351.358169:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1277 00000100:00080000:1.0:1789761351.358178:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761351.358183:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761351.358217:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504036480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761351.358239:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761351.358243:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761351.358245:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761351.358249:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1277 00000100:00080000:1.0:1789761351.358252:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761351.358256:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761351.358265:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761351.358278:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504036608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761351.358286:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504036480:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761351.358289:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761351.358293:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761351.358294:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761351.358297:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1277 00000100:00080000:1.0:1789761351.358300:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761351.358303:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761351.358308:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761351.358336:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504036736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761351.358344:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761351.358347:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761351.358349:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761351.358352:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1277 00000100:00080000:1.0:1789761351.358355:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761351.358357:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761351.358358:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761351.358363:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761351.358372:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504036864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761351.358391:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761351.358394:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761351.358395:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761351.358398:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1277 00000100:00080000:1.0:1789761351.358401:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761351.358419:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761351.358424:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761351/real 1789761351] req@ffff9018c66a5880 x1876699504036480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789761351.358424:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761351.358434:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504036992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761351.358443:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504036480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761351.358451:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761351.358454:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761351.358454:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761351.358456:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761351.358458:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789761351.358460:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1277 00000100:00100000:0.0:1789761351.358462:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504036480:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761351.358463:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761351.358467:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761351.358471:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761351.358479:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504037120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761351.358485:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504036608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761351.358508:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761351.358523:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761351/real 1789761351] req@ffff9018c66a6a00 x1876699504036608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761351.358537:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504036608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761351.358546:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761351.358549:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761351.358553:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504036608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761351.358559:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504036736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761351.358581:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761351.358592:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761351/real 1789761351] req@ffff9018c66a4a80 x1876699504036736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761351.358601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504036736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761351.358606:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761351.358608:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761351.358610:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504036736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761351.358614:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504036864:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761351.358627:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761351.358633:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761351/real 1789761351] req@ffff9018c66a7480 x1876699504036864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761351.358640:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504036864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761351.358646:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761351.358647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761351.358649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504036864:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761351.358652:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504036992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761351.358663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761351.358669:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761351/real 1789761351] req@ffff9018c66a7100 x1876699504036992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761351.358676:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504036992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761351.358681:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761351.358683:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761351.358684:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504036992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761351.358697:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504037120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761351.358707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761351.358713:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761351/real 1789761351] req@ffff9018c66a4380 x1876699504037120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761351.358720:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504037120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761351.358733:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761351.358735:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761351.358737:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504037120:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761351.998198:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761353.022159:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761354.046229:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761355.070180:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761356.094178:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761356.478113:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761356.478123:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761356.478127:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761356.478137:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1283 00000100:00080000:3.0:1789761356.478148:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761356.478156:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761356.478191:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504037248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761356.478217:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761356.478221:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761356.478224:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761356.478228:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1283 00000100:00080000:3.0:1789761356.478232:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761356.478237:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761356.478247:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761356.478259:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504037376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761356.478265:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504037248:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761356.478279:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761356.478283:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761356.478285:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761356.478289:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1283 00000100:00080000:3.0:1789761356.478292:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761356.478296:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761356.478301:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761356.478313:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504037504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761356.478323:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761356.478327:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761356.478329:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761356.478336:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1283 00000800:00000100:0.0:1789761356.478337:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761356.478345:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761356.478350:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761356.478357:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761356.478384:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504037632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761356.478394:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761356/real 1789761356] req@ffff9018d070fb80 x1876699504037248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761356.478399:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761356.478407:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761356.478412:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761356.478419:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1283 00000100:00100000:0.0:1789761356.478424:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504037248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761356.478425:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761356.478435:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789761356.478437:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761356.478443:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:3.0:1789761356.478444:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761356.478449:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504037248:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761356.478462:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504037760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761356.478474:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504037376:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761356.478475:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761356.478483:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761356.478485:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761356.478490:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1283 00000100:00080000:3.0:1789761356.478495:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761356.478499:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789761356.478502:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789761356.478507:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761356.478516:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761356/real 1789761356] req@ffff9018d070c000 x1876699504037376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761356.478523:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504037888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761356.478532:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504037376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761356.478544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761356.478547:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761356.478550:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504037376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761356.478555:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504037504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761356.478569:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761356.478576:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761356/real 1789761356] req@ffff9018d070dc00 x1876699504037504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761356.478584:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504037504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761356.478593:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761356.478596:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761356.478598:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504037504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761356.478601:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504037632:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761356.478612:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761356.478618:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761356/real 1789761356] req@ffff9018d070e300 x1876699504037632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761356.478624:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504037632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761356.478629:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761356.478632:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761356.478633:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504037632:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761356.478644:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504037760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761356.478654:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761356.478660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761356/real 1789761356] req@ffff9018d070d180 x1876699504037760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761356.478667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504037760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761356.478672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761356.478674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761356.478676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504037760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761356.478680:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504037888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761356.478690:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761356.478695:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761356/real 1789761356] req@ffff9018d070f800 x1876699504037888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761356.478702:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504037888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761356.478708:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761356.478709:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761356.478711:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504037888:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761357.118225:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761358.142244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761359.166215:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761360.190167:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761361.214230:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761361.598122:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761361.598130:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761361.598134:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761361.598142:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1288 00000100:00080000:3.0:1789761361.598150:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761361.598156:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761361.598188:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504038016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761361.598210:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761361.598213:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761361.598214:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761361.598217:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1288 00000100:00080000:3.0:1789761361.598219:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761361.598221:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761361.598228:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761361.598236:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504038144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761361.598243:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761361.598246:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761361.598247:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761361.598250:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1288 00000100:00080000:3.0:1789761361.598252:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761361.598254:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761361.598256:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761361.598263:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504038272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761361.598270:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504038016:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761361.598273:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761361.598279:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761361.598281:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761361.598284:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1288 00000100:00080000:3.0:1789761361.598286:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761361.598288:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761361.598292:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761361.598301:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504038400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761361.598308:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761361.598311:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761361.598312:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761361.598314:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1288 00000100:00080000:3.0:1789761361.598317:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761361.598320:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761361.598322:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761361.598392:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504038144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761361.598434:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504038272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761361.598451:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504038528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761361.598461:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761361.598464:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789761361.598465:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504038400:192.168.203.158@tcp:8:kworker.0 00000100:00080000:3.0:1789761361.598466:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761361.598475:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1288 00000100:00080000:3.0:1789761361.598480:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761361.598484:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761361.598489:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761361.598498:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504038528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761361.598502:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504038656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761361.598530:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504038656:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761362.238175:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761363.262175:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761364.286183:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789761364.670376:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789761364.670421:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789761364.670440:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761364.670487:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761364.670502:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761364.670515:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761364.670528:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789761364.670540:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761361/real 1789761364] req@ffff9018d070d500 x1876699504038272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1789761364.670542:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789761364.670574:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504038272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761364.670588:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761364.670604:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761364.670611:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504038272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761364.670620:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761361/real 1789761364] req@ffff9018d070f800 x1876699504038400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761364.670636:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504038400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761364.670645:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761364.670650:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761364.670653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504038400:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761364.670660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761361/real 1789761364] req@ffff9018d070f100 x1876699504038016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761364.670673:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504038016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761364.670681:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761364.670685:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761364.670688:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504038016:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761364.670695:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761361/real 1789761364] req@ffff9018d070e680 x1876699504038144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761364.670710:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504038144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761364.670719:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761364.670723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761364.670727:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504038144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761364.670762:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761361/real 1789761364] req@ffff9018d070e300 x1876699504038656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761364.670776:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504038656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761364.670784:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761364.670788:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761364.670792:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504038656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761364.670799:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761361/real 1789761364] req@ffff9018d070d180 x1876699504038528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761364.670810:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504038528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761364.670819:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761364.670823:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761364.670826:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504038528:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761365.310207:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761366.334215:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761366.718240:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761366.718253:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761366.718258:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761366.718270:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1293 00000100:00080000:3.0:1789761366.718281:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761366.718304:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761366.718349:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504038784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761366.718386:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761366.718391:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761366.718393:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761366.718398:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1293 00000100:00080000:3.0:1789761366.718402:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761366.718407:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761366.718420:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761366.718436:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504038912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761366.718450:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504038784:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761366.718462:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761366.718467:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761366.718469:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761366.718476:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1293 00000100:00080000:3.0:1789761366.718480:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761366.718485:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761366.718491:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761366.718507:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504039040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761366.718516:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761366.718520:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761366.718522:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761366.718526:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1293 00000100:00080000:3.0:1789761366.718542:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761366.718548:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761366.718557:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789761366.718559:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761366.718573:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504039168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761366.718585:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761366.718589:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761366.718590:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761366.718594:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1293 00000100:00080000:3.0:1789761366.718599:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761366.718604:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761366.718609:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761366.718622:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504039296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761366.718630:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761366/real 1789761366] req@ffff9018d070dc00 x1876699504038784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761366.718636:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761366.718646:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761366.718652:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761366.718660:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1293 00000100:00080000:3.0:1789761366.718665:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761366.718669:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504038784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761366.718674:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761366.718684:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761366.718691:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761366.718701:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761366.718701:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504039424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761366.718709:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504038784:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761366.718749:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504038912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761366.718777:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761366.718794:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761366/real 1789761366] req@ffff9018d070c000 x1876699504038912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761366.718809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504038912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761366.718818:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761366.718822:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761366.718826:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504038912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761366.718832:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504039040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761366.718855:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761366.718868:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761366/real 1789761366] req@ffff9018d070fb80 x1876699504039040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761366.718888:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504039040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761366.718897:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761366.718900:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761366.718903:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504039040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761366.718909:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504039168:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761366.718938:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761366.718950:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761366/real 1789761366] req@ffff9018d070d880 x1876699504039168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761366.718961:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504039168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761366.718970:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761366.718974:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761366.718978:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504039168:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761366.718984:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504039296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761366.719006:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761366.719018:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761366/real 1789761366] req@ffff9018d070ce00 x1876699504039296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761366.719032:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504039296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761366.719085:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761366.719088:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761366.719092:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504039296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761366.719098:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504039424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761366.719121:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761366.719134:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761366/real 1789761366] req@ffff9018d070ca80 x1876699504039424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761366.719146:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504039424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761366.719154:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761366.719158:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761366.719163:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504039424:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761367.358218:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761368.382152:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761369.406203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761370.430186:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761371.454264:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761371.838126:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761371.838135:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761371.838139:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761371.838150:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1298 00000100:00080000:0.0:1789761371.838162:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761371.838170:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761371.838207:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504039552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761371.838239:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761371.838243:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761371.838245:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761371.838261:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1298 00000100:00080000:0.0:1789761371.838266:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761371.838270:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761371.838278:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761371.838292:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504039552:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761371.838293:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504039680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761371.838309:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761371.838312:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761371.838314:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761371.838317:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1298 00000100:00080000:0.0:1789761371.838320:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761371.838322:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761371.838326:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761371.838336:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504039808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761371.838343:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761371.838346:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761371.838347:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761371.838351:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1298 00000800:00000100:1.0:1789761371.838355:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761371.838356:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761371.838362:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761371.838366:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761371.838374:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504039936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761371.838381:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761371.838384:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761371.838385:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761371.838390:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1298 00000100:00100000:1.0:1789761371.838390:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761371/real 1789761371] req@ffff9018c8b9a680 x1876699504039552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761371.838393:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761371.838399:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761371.838406:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761371.838414:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504039552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761371.838416:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504040064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761371.838441:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761371.838441:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761371.838448:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761371.838451:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761371.838451:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761371.838458:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504039552:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761371.838459:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1298 00000100:00080000:0.0:1789761371.838467:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761371.838471:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761371.838476:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761371.838484:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504039680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761371.838500:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504040192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761371.838516:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761371.838535:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761371/real 1789761371] req@ffff9018c8b98000 x1876699504039680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761371.838546:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504039680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761371.838552:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761371.838554:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761371.838557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504039680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761371.838561:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504039808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761371.838576:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761371.838583:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761371/real 1789761371] req@ffff9018c8b9aa00 x1876699504039808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761371.838590:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504039808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761371.838595:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761371.838597:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761371.838599:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504039808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761371.838603:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504039936:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761371.838613:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761371.838619:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761371/real 1789761371] req@ffff9018c8b9b800 x1876699504039936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761371.838631:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504039936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761371.838636:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761371.838637:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761371.838640:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504039936:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761371.838645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504040064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761371.838664:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761371.838676:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761371/real 1789761371] req@ffff9018c8b9b480 x1876699504040064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761371.838689:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504040064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761371.838698:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761371.838701:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761371.838703:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504040064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761371.838720:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504040192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761371.838737:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761371.838745:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761371/real 1789761371] req@ffff9018c8b99f80 x1876699504040192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761371.838753:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504040192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761371.838763:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761371.838766:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761371.838769:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504040192:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761372.478152:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761373.502184:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761374.526148:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761375.550177:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761376.574202:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761376.958139:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761376.958148:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761376.958151:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761376.958159:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1303 00000100:00080000:0.0:1789761376.958169:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761376.958174:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761376.958230:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504040320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761376.958262:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761376.958266:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761376.958268:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761376.958273:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1303 00000100:00080000:0.0:1789761376.958276:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761376.958281:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761376.958291:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761376.958306:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504040448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761376.958309:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504040320:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761376.958325:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761376.958328:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761376.958330:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761376.958335:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1303 00000100:00080000:0.0:1789761376.958338:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761376.958342:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761376.958348:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761376.958360:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504040576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761376.958370:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761376.958372:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761376.958377:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761376.958379:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761376.958384:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1303 00000100:00080000:0.0:1789761376.958389:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761376.958393:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761376.958400:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761376.958411:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504040704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761376.958414:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761376/real 1789761376] req@ffff9018c8b9bb80 x1876699504040320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761376.958430:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761376.958434:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761376.958436:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761376.958441:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1303 00000100:00100000:1.0:1789761376.958442:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504040320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761376.958444:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761376.958451:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761376.958458:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761376.958468:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789761376.958469:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504040832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761376.958480:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761376.958481:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761376.958486:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761376.958488:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761376.958490:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504040320:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761376.958493:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1303 00000100:00080000:0.0:1789761376.958497:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761376.958502:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761376.958504:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761376.958512:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504040960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761376.958522:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504040448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761376.958544:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761376.958553:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761376/real 1789761376] req@ffff9018c8b98700 x1876699504040448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761376.958561:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504040448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761376.958567:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761376.958569:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761376.958572:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504040448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761376.958576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504040576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761376.958586:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761376.958592:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761376/real 1789761376] req@ffff9018c8b98a80 x1876699504040576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761376.958599:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504040576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761376.958604:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761376.958606:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761376.958607:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504040576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761376.958611:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504040704:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761376.958621:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761376.958627:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761376/real 1789761376] req@ffff9018c8b98e00 x1876699504040704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761376.958633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504040704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761376.958639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761376.958640:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761376.958642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504040704:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761376.958646:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504040832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761376.958655:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761376.958661:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761376/real 1789761376] req@ffff9018c8b99880 x1876699504040832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761376.958668:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504040832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761376.958674:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761376.958675:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761376.958677:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504040832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761376.958687:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504040960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761376.958707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761376.958713:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761376/real 1789761376] req@ffff9018c8b99f80 x1876699504040960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761376.958720:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504040960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761376.958728:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761376.958730:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761376.958732:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504040960:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761377.598232:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761378.622238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761379.646242:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761380.670247:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761381.694213:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761382.078227:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761382.078238:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761382.078242:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761382.078254:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1308 00000100:00080000:0.0:1789761382.078267:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761382.078275:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761382.078317:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504041088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761382.078340:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761382.078343:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761382.078345:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761382.078348:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1308 00000100:00080000:0.0:1789761382.078351:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761382.078353:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761382.078364:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761382.078376:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504041216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761382.078386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761382.078388:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761382.078391:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761382.078397:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1308 00000100:00080000:0.0:1789761382.078403:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761382.078404:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504041088:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761382.078412:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761382.078427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761382.078445:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504041344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761382.078453:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761382.078456:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761382.078458:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761382.078470:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1308 00000100:00080000:0.0:1789761382.078476:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761382.078479:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761382.078484:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789761382.078485:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761382.078500:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504041472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761382.078510:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761382.078514:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761382.078516:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761382.078519:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1308 00000100:00080000:0.0:1789761382.078522:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761382.078525:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761382.078529:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761382.078541:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504041600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761382.078549:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761382.078552:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761382/real 1789761382] req@ffff9018c8b9b480 x1876699504041088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761382.078553:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761382.078558:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761382.078569:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1308 00000100:00080000:0.0:1789761382.078574:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761382.078582:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761382.078590:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761382.078608:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504041728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761382.078609:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504041088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761382.078678:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761382.078683:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761382.078689:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504041088:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761382.078711:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504041216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761382.078731:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761382.078740:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761382/real 1789761382] req@ffff9018c8b9b800 x1876699504041216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761382.078754:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504041216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761382.078761:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761382.078763:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761382.078765:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504041216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761382.078769:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504041344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761382.078782:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761382.078788:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761382/real 1789761382] req@ffff9018c8b9aa00 x1876699504041344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761382.078796:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504041344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761382.078801:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761382.078803:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761382.078806:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504041344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761382.078810:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504041472:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761382.078821:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761382.078827:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761382/real 1789761382] req@ffff9018c8b98000 x1876699504041472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761382.078834:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504041472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761382.078840:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761382.078841:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761382.078843:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504041472:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761382.078849:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504041600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761382.078870:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761382.078882:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761382/real 1789761382] req@ffff9018c8b9a680 x1876699504041600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761382.078894:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504041600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761382.078903:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761382.078907:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761382.078910:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504041600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761382.078918:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504041728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761382.078939:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761382.078948:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761382/real 1789761382] req@ffff9018c8b9b100 x1876699504041728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761382.078956:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504041728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761382.078962:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761382.078964:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761382.078966:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504041728:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761382.718195:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761383.742205:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761384.766227:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761385.790228:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761386.814205:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761387.198162:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761387.198170:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761387.198174:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761387.198182:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1313 00000100:00080000:3.0:1789761387.198192:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761387.198200:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761387.198239:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504041856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761387.198263:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761387.198266:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761387.198267:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761387.198271:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1313 00000100:00080000:3.0:1789761387.198274:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761387.198277:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761387.198285:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761387.198296:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504041984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761387.198304:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761387.198306:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761387.198308:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761387.198312:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504041856:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761387.198313:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1313 00000100:00080000:3.0:1789761387.198318:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761387.198322:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761387.198325:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761387.198333:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504042112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761387.198339:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761387.198342:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761387.198343:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761387.198346:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1313 00000100:00080000:3.0:1789761387.198348:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761387.198351:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761387.198354:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761387.198363:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504042240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761387.198369:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761387.198373:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789761387.198375:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761387.198375:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761387.198381:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1313 00000100:00080000:3.0:1789761387.198384:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761387.198388:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761387.198391:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761387.198399:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504042368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761387.198405:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761387.198408:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761387.198409:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761387.198411:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1313 00000100:00080000:3.0:1789761387.198413:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761387.198416:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761387.198420:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761387.198424:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761387/real 1789761387] req@ffff9018d070df80 x1876699504041856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761387.198431:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504042496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761387.198445:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504041856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761387.198455:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761387.198458:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761387.198462:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504041856:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761387.198482:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504041984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761387.198499:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761387.198508:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761387/real 1789761387] req@ffff9018d070ed80 x1876699504041984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761387.198515:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504041984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761387.198525:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761387.198527:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761387.198530:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504041984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761387.198533:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504042112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761387.198543:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761387.198549:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761387/real 1789761387] req@ffff9018d070f480 x1876699504042112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761387.198556:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504042112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761387.198561:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761387.198563:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761387.198565:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504042112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761387.198568:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504042240:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761387.198578:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761387.198584:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761387/real 1789761387] req@ffff9018d070ca80 x1876699504042240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761387.198591:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504042240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761387.198596:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761387.198598:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761387.198600:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504042240:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761387.198604:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504042368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761387.198622:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761387.198629:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761387/real 1789761387] req@ffff9018d070ce00 x1876699504042368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761387.198636:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504042368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761387.198641:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761387.198643:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761387.198646:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504042368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761387.198649:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504042496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761387.198660:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761387.198665:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761387/real 1789761387] req@ffff9018d070d880 x1876699504042496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761387.198672:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504042496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761387.198677:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761387.198679:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761387.198680:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504042496:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761387.838160:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761388.862228:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761389.886185:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761390.910252:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761391.934215:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761392.318131:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761392.318139:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761392.318142:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761392.318151:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1318 00000100:00080000:3.0:1789761392.318159:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761392.318165:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761392.318196:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504042624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761392.318219:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761392.318222:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761392.318223:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761392.318226:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1318 00000100:00080000:3.0:1789761392.318228:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761392.318230:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761392.318238:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761392.318246:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504042752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761392.318253:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761392.318256:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761392.318258:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761392.318265:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1318 00000100:00080000:3.0:1789761392.318269:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761392.318272:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761392.318276:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761392.318282:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504042880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761392.318288:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761392.318290:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761392.318291:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761392.318295:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1318 00000100:00080000:3.0:1789761392.318297:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761392.318299:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761392.318302:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761392.318315:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504043008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761392.318322:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504042624:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761392.318324:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761392.318338:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761392.318340:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761392.318344:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1318 00000100:00080000:3.0:1789761392.318346:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761392.318348:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761392.318350:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761392.318365:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504043136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761392.318370:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761392.318373:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761392.318374:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761392.318377:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1318 00000100:00080000:3.0:1789761392.318379:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761392.318382:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761392.318384:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761392.318391:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504043264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761392.318418:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761392.318473:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761392/real 1789761392] req@ffff9018d070fb80 x1876699504042624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761392.318492:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504042624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761392.318503:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761392.318507:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761392.318511:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504042624:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761392.318517:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504042752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761392.318533:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761392.318542:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761392/real 1789761392] req@ffff9018d070c000 x1876699504042752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761392.318550:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504042752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761392.318555:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761392.318557:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761392.318559:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504042752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761392.318563:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504042880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761392.318574:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761392.318580:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761392/real 1789761392] req@ffff9018d070dc00 x1876699504042880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761392.318592:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504042880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761392.318607:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761392.318610:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761392.318614:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504042880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761392.318648:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504043008:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761392.318665:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761392.318674:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761392/real 1789761392] req@ffff9018d070d180 x1876699504043008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761392.318682:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504043008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761392.318688:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761392.318690:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761392.318692:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504043008:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761392.318696:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504043136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761392.318706:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761392.318711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761392/real 1789761392] req@ffff9018d070e300 x1876699504043136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761392.318719:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504043136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761392.318723:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761392.318725:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761392.318728:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504043136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761392.318732:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504043264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761392.318742:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761392.318748:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761392/real 1789761392] req@ffff9018d070e680 x1876699504043264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761392.318755:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504043264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761392.318760:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761392.318762:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761392.318764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504043264:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761392.958249:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761393.982203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761395.006192:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761396.030212:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761397.054181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761397.438169:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761397.438180:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761397.438185:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761397.438196:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1323 00000100:00080000:2.0:1789761397.438208:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761397.438214:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761397.438258:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504043392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761397.438293:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761397.438296:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761397.438297:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761397.438301:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1323 00000100:00080000:2.0:1789761397.438303:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761397.438307:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761397.438317:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761397.438327:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504043520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761397.438335:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761397.438338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761397.438340:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761397.438343:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1323 00000100:00080000:2.0:1789761397.438345:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761397.438350:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761397.438351:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504043392:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789761397.438353:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761397.438364:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504043648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761397.438371:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761397.438374:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761397.438375:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761397.438377:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1323 00000100:00080000:2.0:1789761397.438379:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761397.438382:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761397.438385:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761397.438394:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504043776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761397.438401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761397.438405:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761397.438406:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761397.438409:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1323 00000100:00080000:2.0:1789761397.438415:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761397.438418:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761397.438421:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761397.438428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504043904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761397.438436:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761397.438439:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761397.438443:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761397.438445:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761397.438448:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1323 00000100:00080000:2.0:1789761397.438451:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761397.438454:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761397.438457:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761397.438464:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504044032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761397.438497:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761397/real 1789761397] req@ffff9018d0f27800 x1876699504043392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761397.438516:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504043392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761397.438526:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761397.438529:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761397.438533:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504043392:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761397.438539:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504043520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761397.438556:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761397.438565:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761397/real 1789761397] req@ffff9018d0f25180 x1876699504043520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761397.438579:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504043520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761397.438585:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761397.438588:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761397.438590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504043520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761397.438618:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504043648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761397.438640:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761397.438653:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761397/real 1789761397] req@ffff9018d0f24a80 x1876699504043648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761397.438667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504043648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761397.438676:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761397.438680:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761397.438683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504043648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761397.438690:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504043776:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761397.438724:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761397.438737:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761397/real 1789761397] req@ffff9018d0f24380 x1876699504043776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761397.438751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504043776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761397.438760:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761397.438764:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761397.438767:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504043776:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761397.438774:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504043904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761397.438797:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761397.438810:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761397/real 1789761397] req@ffff9018d0f25500 x1876699504043904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761397.438823:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504043904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761397.438832:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761397.438845:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761397.438852:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504043904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761397.438860:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504044032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761397.438883:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761397.438897:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761397/real 1789761397] req@ffff9018d0f24000 x1876699504044032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761397.438910:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504044032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761397.438920:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761397.438923:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761397.438927:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504044032:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761398.078216:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761399.102252:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761400.126230:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761401.150202:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761402.174172:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761402.558091:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761402.558099:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761402.558102:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761402.558110:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1329 00000100:00080000:2.0:1789761402.558119:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761402.558125:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761402.558160:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504044160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761402.558181:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761402.558190:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761402.558191:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761402.558194:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1329 00000100:00080000:2.0:1789761402.558197:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761402.558200:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761402.558208:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761402.558217:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504044288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761402.558225:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761402.558229:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789761402.558230:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504044160:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761402.558231:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761402.558239:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1329 00000100:00080000:2.0:1789761402.558243:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761402.558245:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761402.558248:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761402.558253:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504044416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761402.558260:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761402.558264:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761402.558267:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761402.558271:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1329 00000100:00080000:2.0:1789761402.558275:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761402.558280:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761402.558286:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761402.558299:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504044544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761402.558309:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761402.558312:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761402.558317:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761402.558320:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761402.558325:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1329 00000100:00080000:2.0:1789761402.558330:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761402.558335:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761402.558341:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761402.558357:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761402/real 1789761402] req@ffff9018d0f24700 x1876699504044160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761402.558358:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504044672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761402.558396:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761402.558401:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504044160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761402.558402:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761402.558406:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761402.558417:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1329 00000100:00080000:1.0:1789761402.558422:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761402.558424:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761402.558432:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789761402.558432:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761402.558441:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761402.558442:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504044160:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761402.558461:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504044800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761402.558470:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504044288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761402.558498:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761402.558512:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761402/real 1789761402] req@ffff9018d0f26680 x1876699504044288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761402.558526:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504044288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761402.558535:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761402.558539:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761402.558543:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504044288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761402.558549:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504044416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761402.558567:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761402.558578:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761402/real 1789761402] req@ffff9018d0f24e00 x1876699504044416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761402.558591:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504044416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761402.558601:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761402.558604:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761402.558608:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504044416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761402.558615:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504044544:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761402.558633:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761402.558644:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761402/real 1789761402] req@ffff9018d0f27480 x1876699504044544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761402.558657:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504044544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761402.558667:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761402.558671:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761402.558674:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504044544:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761402.558681:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504044672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761402.558698:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761402.558708:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761402/real 1789761402] req@ffff9018d0f24000 x1876699504044672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761402.558721:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504044672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761402.558731:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761402.558735:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761402.558738:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504044672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761402.558756:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504044800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761402.558775:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761402.558786:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761402/real 1789761402] req@ffff9018d0f25500 x1876699504044800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761402.558800:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504044800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761402.558809:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761402.558814:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761402.558817:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504044800:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761403.198213:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761404.222195:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761405.246160:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761406.270157:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761407.294197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761407.678146:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761407.678155:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761407.678159:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761407.678171:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1334 00000100:00080000:2.0:1789761407.678182:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761407.678190:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761407.678224:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504044928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761407.678250:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761407.678253:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761407.678254:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761407.678258:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1334 00000100:00080000:2.0:1789761407.678266:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761407.678271:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761407.678280:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761407.678290:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504044928:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761407.678297:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504045056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761407.678307:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761407.678310:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761407.678311:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761407.678315:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1334 00000100:00080000:2.0:1789761407.678318:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761407.678322:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761407.678326:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761407.678337:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504045184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761407.678343:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761407.678345:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761407.678347:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761407.678351:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1334 00000100:00080000:2.0:1789761407.678355:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761407.678359:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761407.678364:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761407.678372:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761407.678387:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504045312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761407.678398:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761407.678403:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761407.678405:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761407.678410:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1334 00000100:00080000:2.0:1789761407.678415:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761407.678418:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761407/real 1789761407] req@ffff9018d0f24380 x1876699504044928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761407.678423:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761407.678433:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761407.678449:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504045440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761407.678451:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504044928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761407.678475:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761407.678478:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761407.678480:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761407.678485:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761407.678488:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789761407.678492:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1334 00000100:00080000:2.0:1789761407.678497:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761407.678501:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761407.678505:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761407.678506:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504044928:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761407.678516:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504045568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761407.678536:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504045056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761407.678560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761407.678575:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761407/real 1789761407] req@ffff9018d0f24a80 x1876699504045056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761407.678590:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504045056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761407.678600:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761407.678605:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761407.678609:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504045056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761407.678617:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504045184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761407.678641:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761407.678654:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761407/real 1789761407] req@ffff9018d0f25180 x1876699504045184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761407.678667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504045184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761407.678674:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761407.678676:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761407.678679:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504045184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761407.678683:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504045312:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761407.678699:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761407.678709:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761407/real 1789761407] req@ffff9018d0f27800 x1876699504045312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761407.678717:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504045312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761407.678723:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761407.678725:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761407.678727:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504045312:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761407.678731:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504045440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761407.678755:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761407.678768:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761407/real 1789761407] req@ffff9018d0f26d80 x1876699504045440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761407.678785:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504045440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761407.678795:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761407.678800:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761407.678804:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504045440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761407.678813:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504045568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761407.678841:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761407.678855:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761407/real 1789761407] req@ffff9018d0f27100 x1876699504045568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761407.678864:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504045568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761407.678873:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761407.678876:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761407.678879:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504045568:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761408.318213:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761409.342173:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761410.366244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761411.390174:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761412.414228:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761412.798153:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761412.798168:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761412.798173:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761412.798184:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1339 00000100:00080000:0.0:1789761412.798196:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761412.798204:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761412.798254:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504045696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761412.798287:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761412.798292:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761412.798294:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761412.798298:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1339 00000100:00080000:0.0:1789761412.798303:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761412.798307:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761412.798320:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761412.798334:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504045824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761412.798348:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504045696:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761412.798349:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761412.798359:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761412.798361:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761412.798366:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1339 00000100:00080000:0.0:1789761412.798370:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761412.798374:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761412.798379:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761412.798390:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504045952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761412.798407:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761412.798411:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761412.798413:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761412.798417:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1339 00000100:00080000:0.0:1789761412.798421:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761412.798425:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761412.798430:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789761412.798438:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761412.798443:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504046080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761412.798455:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761412.798460:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761412.798462:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761412.798466:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1339 00000100:00080000:0.0:1789761412.798471:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761412.798475:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761412.798481:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761412.798491:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504046208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761412.798499:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761412.798503:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789761412.798504:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761412/real 1789761412] req@ffff9018c8b98380 x1876699504045696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761412.798505:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761412.798512:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1339 00000100:00080000:0.0:1789761412.798518:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761412.798524:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761412.798529:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761412.798539:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504045696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761412.798540:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504046336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761412.798577:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761412.798585:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761412.798590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504045696:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761412.798620:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504045824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761412.798662:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761412.798678:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761412/real 1789761412] req@ffff9018c8b99c00 x1876699504045824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761412.798693:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504045824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761412.798702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761412.798706:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761412.798710:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504045824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761412.798717:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504045952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761412.798742:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761412.798755:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761412/real 1789761412] req@ffff9018c8b99500 x1876699504045952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761412.798768:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504045952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761412.798777:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761412.798781:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761412.798785:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504045952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761412.798791:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504046080:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761412.798815:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761412.798828:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761412/real 1789761412] req@ffff9018c8b9ad80 x1876699504046080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761412.798842:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504046080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761412.798851:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761412.798855:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761412.798858:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504046080:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761412.798865:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504046208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761412.798884:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761412.798892:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761412/real 1789761412] req@ffff9018c8b9a300 x1876699504046208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761412.798900:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504046208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761412.798905:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761412.798908:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761412.798909:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504046208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761412.798913:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504046336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761412.798923:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761412.798929:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761412/real 1789761412] req@ffff9018c8b9b100 x1876699504046336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761412.798936:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504046336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761412.798941:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761412.798956:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761412.798960:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504046336:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761413.438219:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761414.462228:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761415.486176:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761416.510204:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761417.534180:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761417.918125:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761417.918133:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761417.918138:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761417.918146:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1344 00000100:00080000:0.0:1789761417.918165:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761417.918171:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761417.918208:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504046464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761417.918232:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761417.918235:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761417.918236:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761417.918239:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1344 00000100:00080000:0.0:1789761417.918241:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761417.918244:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761417.918251:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761417.918260:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504046592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761417.918268:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761417.918271:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761417.918273:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761417.918276:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1344 00000100:00080000:0.0:1789761417.918278:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761417.918282:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761417.918284:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504046464:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789761417.918286:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761417.918296:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504046720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761417.918304:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761417.918306:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761417.918307:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761417.918309:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1344 00000100:00080000:0.0:1789761417.918312:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761417.918314:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761417.918318:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761417.918325:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504046848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761417.918332:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761417.918334:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761417.918336:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761417.918338:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1344 00000100:00080000:0.0:1789761417.918342:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761417.918342:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761417.918347:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761417.918352:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761417.918360:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504046976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761417.918367:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761417.918369:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761417.918370:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761417.918373:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1344 00000100:00080000:0.0:1789761417.918375:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761417.918376:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761417.918379:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761417.918385:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504047104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761417.918398:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761417/real 1789761417] req@ffff9018c8b9a680 x1876699504046464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761417.918422:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504046464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761417.918440:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761417.918443:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761417.918446:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504046464:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761417.918452:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504046592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761417.918472:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761417.918481:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761417/real 1789761417] req@ffff9018c8b98000 x1876699504046592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761417.918490:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504046592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761417.918495:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761417.918498:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761417.918500:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504046592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761417.918518:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504046720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761417.918529:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761417.918535:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761417/real 1789761417] req@ffff9018c8b9aa00 x1876699504046720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761417.918543:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504046720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761417.918548:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761417.918550:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761417.918553:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504046720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761417.918557:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504046848:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761417.918567:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761417.918577:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761417/real 1789761417] req@ffff9018c8b9b800 x1876699504046848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761417.918595:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504046848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761417.918605:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761417.918608:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761417.918612:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504046848:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761417.918619:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504046976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761417.918640:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761417.918653:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761417/real 1789761417] req@ffff9018c8b9b480 x1876699504046976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761417.918665:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504046976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761417.918674:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761417.918678:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761417.918681:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504046976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761417.918685:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504047104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761417.918701:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761417.918708:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761417/real 1789761417] req@ffff9018c8b99f80 x1876699504047104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761417.918717:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504047104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761417.918722:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761417.918724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761417.918726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504047104:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761418.558252:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761419.582194:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761420.606240:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761421.630210:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761422.654173:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761423.038149:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761423.038157:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761423.038160:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761423.038169:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1349 00000100:00080000:3.0:1789761423.038179:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761423.038184:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761423.038218:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504047232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761423.038237:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761423.038240:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761423.038241:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761423.038244:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1349 00000100:00080000:3.0:1789761423.038247:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761423.038249:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761423.038257:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761423.038265:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504047360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761423.038272:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761423.038275:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761423.038276:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761423.038279:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1349 00000100:00080000:3.0:1789761423.038280:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761423.038283:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761423.038286:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761423.038295:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504047488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761423.038296:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504047232:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761423.038309:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761423.038312:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761423.038313:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761423.038315:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1349 00000100:00080000:3.0:1789761423.038317:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761423.038319:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761423.038323:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761423.038331:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504047616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761423.038338:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761423.038340:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761423.038341:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761423.038343:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1349 00000100:00080000:3.0:1789761423.038346:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761423.038348:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761423.038357:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761423.038368:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504047744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761423.038376:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761423.038378:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761423.038381:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761423.038383:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761423.038386:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1349 00000100:00080000:3.0:1789761423.038389:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761423.038391:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761423.038395:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761423.038402:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504047872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761423.038429:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761423/real 1789761423] req@ffff9018d070f100 x1876699504047232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761423.038445:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504047232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761423.038454:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761423.038457:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761423.038461:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504047232:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761423.038467:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504047360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761423.038488:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761423.038502:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761423/real 1789761423] req@ffff9018d070f800 x1876699504047360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761423.038516:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504047360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761423.038533:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761423.038537:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761423.038541:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504047360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761423.038567:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504047488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761423.038590:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761423.038603:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761423/real 1789761423] req@ffff9018d070d500 x1876699504047488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761423.038616:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504047488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761423.038625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761423.038629:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761423.038633:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504047488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761423.038640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504047616:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761423.038661:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761423.038671:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761423/real 1789761423] req@ffff9018d070e680 x1876699504047616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761423.038679:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504047616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761423.038685:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761423.038687:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761423.038690:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504047616:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761423.038694:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504047744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761423.038706:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761423.038712:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761423/real 1789761423] req@ffff9018d070e300 x1876699504047744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761423.038720:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504047744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761423.038725:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761423.038726:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761423.038728:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504047744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761423.038731:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504047872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761423.038741:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761423.038746:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761423/real 1789761423] req@ffff9018d070d180 x1876699504047872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761423.038753:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504047872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761423.038758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761423.038760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761423.038762:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504047872:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761423.678250:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761424.702252:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761425.726271:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761426.750170:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761427.774245:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761428.158166:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761428.158174:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761428.158177:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761428.158185:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1354 00000100:00080000:0.0:1789761428.158194:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761428.158199:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761428.158229:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504048000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761428.158254:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761428.158257:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761428.158258:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761428.158261:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1354 00000100:00080000:0.0:1789761428.158264:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761428.158266:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761428.158273:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761428.158282:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504048128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761428.158289:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761428.158292:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761428.158293:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761428.158296:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1354 00000100:00080000:0.0:1789761428.158298:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761428.158300:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761428.158303:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761428.158307:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504048000:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761428.158323:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504048256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761428.158336:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761428.158340:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761428.158342:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761428.158347:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1354 00000100:00080000:0.0:1789761428.158353:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761428.158358:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761428.158364:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761428.158374:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504048384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761428.158381:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761428.158384:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761428.158385:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761428.158388:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1354 00000100:00080000:0.0:1789761428.158391:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761428.158394:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761428.158398:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761428.158405:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504048128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761428.158408:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504048512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761428.158420:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761428.158423:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761428.158424:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761428.158427:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1354 00000100:00080000:0.0:1789761428.158435:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761428.158438:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761428.158440:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761428.158451:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504048640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761428.158468:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504048256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761428.158489:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504048384:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761428.158516:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504048512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761428.158530:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504048640:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761428.798199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761429.822196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761430.846169:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789761431.230277:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789761431.230314:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789761431.230327:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761431.230355:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761431.230364:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761431.230374:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761431.230386:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761431.230398:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789761431.230411:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761428/real 1789761431] req@ffff9018c8b98a80 x1876699504048256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761431.230444:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504048256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761431.230458:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761431.230463:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761431.230468:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504048256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761431.230478:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761428/real 1789761431] req@ffff9018c8b98700 x1876699504048384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761431.230494:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504048384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761431.230509:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761431.230514:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761431.230518:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504048384:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761431.230526:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761428/real 1789761431] req@ffff9018c8b9bb80 x1876699504048512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761431.230547:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504048512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761431.230557:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761431.230561:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761431.230565:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504048512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761431.230572:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761428/real 1789761431] req@ffff9018c8b99f80 x1876699504048640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761431.230586:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504048640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761431.230595:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761431.230599:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761431.230603:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504048640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761431.230610:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761428/real 1789761431] req@ffff9018c8b99880 x1876699504048000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761431.230629:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504048000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761431.230639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761431.230644:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761431.230648:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504048000:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761431.230656:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761428/real 1789761431] req@ffff9018c8b98e00 x1876699504048128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761431.230670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504048128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761431.230680:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761431.230684:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761431.230688:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504048128:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761431.870184:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761432.894151:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761433.278161:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761433.278169:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761433.278173:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761433.278182:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1359 00000100:00080000:2.0:1789761433.278191:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761433.278197:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761433.278225:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504048768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761433.278245:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761433.278247:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761433.278248:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761433.278251:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1359 00000100:00080000:2.0:1789761433.278254:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761433.278257:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761433.278267:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761433.278282:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504048896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761433.278291:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504048768:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761433.278296:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761433.278302:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761433.278304:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761433.278309:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1359 00000100:00080000:2.0:1789761433.278313:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761433.278317:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761433.278323:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761433.278337:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504049024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761433.278347:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761433.278350:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761433.278352:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761433.278356:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1359 00000100:00080000:2.0:1789761433.278361:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761433.278362:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761433.278369:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761433.278376:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761433.278405:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504049152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761433.278417:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761433/real 1789761433] req@ffff9018d0f26a00 x1876699504048768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761433.278418:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761433.278426:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761433.278430:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761433.278439:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1359 00000100:00080000:2.0:1789761433.278445:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761433.278450:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504048768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761433.278453:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761433.278462:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761433.278467:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761433.278474:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761433.278479:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504049280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761433.278482:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504048768:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761433.278498:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761433.278502:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761433.278504:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761433.278510:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1359 00000100:00100000:1.0:1789761433.278513:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504048896:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761433.278516:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761433.278525:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761433.278532:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761433.278548:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761433.278552:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504049408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761433.278567:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761433/real 1789761433] req@ffff9018d0f25c00 x1876699504048896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761433.278585:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504048896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761433.278601:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761433.278606:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761433.278610:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504048896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761433.278617:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504049024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761433.278637:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761433.278649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761433/real 1789761433] req@ffff9018d0f25f80 x1876699504049024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761433.278661:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504049024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761433.278673:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761433.278677:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761433.278681:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504049024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761433.278688:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504049152:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761433.278707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761433.278717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761433/real 1789761433] req@ffff9018d0f26300 x1876699504049152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761433.278729:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504049152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761433.278737:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761433.278741:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761433.278745:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504049152:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761433.278751:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504049280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761433.278769:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761433.278780:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761433/real 1789761433] req@ffff9018d0f27100 x1876699504049280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761433.278807:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504049280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761433.278817:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761433.278821:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761433.278825:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504049280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761433.278858:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504049408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761433.278881:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761433.278894:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761433/real 1789761433] req@ffff9018d0f26d80 x1876699504049408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761433.278907:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504049408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761433.278916:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761433.278919:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761433.278922:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504049408:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761433.918168:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761434.942200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761435.966212:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761436.990155:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761438.014206:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761438.398129:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761438.398137:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761438.398140:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761438.398148:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1364 00000100:00080000:0.0:1789761438.398157:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761438.398163:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761438.398198:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504049536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761438.398225:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761438.398229:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761438.398231:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761438.398236:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1364 00000100:00080000:0.0:1789761438.398240:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761438.398244:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761438.398253:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761438.398265:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504049664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761438.398273:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504049536:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761438.398276:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761438.398284:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761438.398286:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761438.398289:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1364 00000100:00080000:0.0:1789761438.398292:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761438.398294:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761438.398298:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761438.398306:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504049792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761438.398313:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761438.398315:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761438.398317:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761438.398319:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1364 00000100:00080000:0.0:1789761438.398330:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761438.398332:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761438.398336:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789761438.398342:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761438.398347:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504049920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761438.398355:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761438.398357:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761438.398359:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761438.398362:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1364 00000100:00080000:0.0:1789761438.398364:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761438.398367:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761438.398370:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761438.398378:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504050048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761438.398384:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761438.398386:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761438.398388:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761438.398391:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1364 00000100:00080000:0.0:1789761438.398393:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761438.398396:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761438.398400:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761438.398405:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761438/real 1789761438] req@ffff9018c8b9b480 x1876699504049536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761438.398412:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504050176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761438.398443:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504049536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761438.398456:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761438.398461:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761438.398466:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504049536:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761438.398497:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504049664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761438.398521:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761438.398535:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761438/real 1789761438] req@ffff9018c8b9b800 x1876699504049664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761438.398549:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504049664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761438.398559:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761438.398562:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761438.398567:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504049664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761438.398573:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504049792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761438.398592:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761438.398602:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761438/real 1789761438] req@ffff9018c8b9aa00 x1876699504049792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761438.398616:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504049792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761438.398633:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761438.398637:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761438.398640:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504049792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761438.398647:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504049920:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761438.398666:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761438.398676:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761438/real 1789761438] req@ffff9018c8b98000 x1876699504049920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761438.398689:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504049920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761438.398699:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761438.398702:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761438.398706:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504049920:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761438.398712:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504050048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761438.398731:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761438.398741:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761438/real 1789761438] req@ffff9018c8b9a680 x1876699504050048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761438.398768:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504050048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761438.398778:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761438.398781:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761438.398786:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504050048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761438.398800:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504050176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761438.398822:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761438.398835:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761438/real 1789761438] req@ffff9018c8b9b100 x1876699504050176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761438.398848:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504050176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761438.398857:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761438.398861:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761438.398865:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504050176:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761439.038195:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761440.062210:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761441.086267:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761442.110193:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761443.134194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761443.518157:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761443.518165:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761443.518169:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761443.518179:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1370 00000100:00080000:3.0:1789761443.518188:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761443.518194:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761443.518227:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504050304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761443.518247:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761443.518251:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761443.518252:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761443.518256:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1370 00000100:00080000:3.0:1789761443.518259:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761443.518263:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761443.518273:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761443.518285:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504050432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761443.518298:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504050304:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761443.518300:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761443.518305:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761443.518308:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761443.518312:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1370 00000100:00080000:3.0:1789761443.518314:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761443.518316:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761443.518323:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761443.518333:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504050560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761443.518340:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761443.518343:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761443.518344:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761443.518348:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1370 00000100:00080000:3.0:1789761443.518352:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761443.518354:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761443.518357:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761443.518365:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504050688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761443.518373:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761443.518374:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761443.518379:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761443.518380:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761443.518383:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1370 00000100:00080000:3.0:1789761443.518389:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761443.518392:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761443.518397:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761443.518412:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504050816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761443.518420:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761443.518424:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761443.518427:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761443.518431:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1370 00000100:00080000:3.0:1789761443.518435:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761443.518439:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761443/real 1789761443] req@ffff9018d070dc00 x1876699504050304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761443.518442:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761443.518450:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761443.518460:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504050944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761443.518464:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504050304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761443.518487:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761443.518491:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761443.518495:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504050304:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761443.518536:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504050432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761443.518560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761443.518571:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761443/real 1789761443] req@ffff9018d070c000 x1876699504050432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761443.518582:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504050432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761443.518588:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761443.518591:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761443.518594:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504050432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761443.518597:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504050560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761443.518609:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761443.518616:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761443/real 1789761443] req@ffff9018d070fb80 x1876699504050560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761443.518623:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504050560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761443.518632:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761443.518635:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761443.518638:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504050560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761443.518644:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504050688:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761443.518665:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761443.518678:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761443/real 1789761443] req@ffff9018d070d880 x1876699504050688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761443.518688:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504050688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761443.518694:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761443.518697:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761443.518700:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504050688:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761443.518704:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504050816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761443.518734:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761443.518743:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761443/real 1789761443] req@ffff9018d070ce00 x1876699504050816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761443.518752:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504050816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761443.518758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761443.518760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761443.518762:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504050816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761443.518767:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504050944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761443.518787:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761443.518810:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761443/real 1789761443] req@ffff9018d070ca80 x1876699504050944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761443.518827:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504050944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761443.518834:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761443.518837:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761443.518840:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504050944:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761444.158218:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761445.182263:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761446.206221:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761447.230281:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761448.254140:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761448.638128:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761448.638135:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761448.638139:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761448.638147:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1375 00000100:00080000:3.0:1789761448.638156:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761448.638163:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761448.638194:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504051072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761448.638223:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761448.638227:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761448.638230:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761448.638234:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1375 00000100:00080000:3.0:1789761448.638238:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761448.638243:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761448.638254:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761448.638266:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504051072:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761448.638271:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504051200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761448.638281:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761448.638284:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761448.638285:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761448.638288:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1375 00000100:00080000:3.0:1789761448.638291:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761448.638294:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761448.638298:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761448.638307:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504051328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761448.638314:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761448.638317:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761448.638318:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761448.638320:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1375 00000100:00080000:3.0:1789761448.638323:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761448.638328:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789761448.638330:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789761448.638333:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761448.638343:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504051456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761448.638350:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761448.638353:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761448.638354:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761448.638356:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1375 00000100:00080000:3.0:1789761448.638358:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761448.638361:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761448.638364:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761448.638372:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504051584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761448.638377:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761448/real 1789761448] req@ffff9018d070f480 x1876699504051072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761448.638383:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761448.638391:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761448.638399:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761448.638403:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504051072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761448.638404:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1375 00000100:00080000:3.0:1789761448.638411:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761448.638417:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761448.638423:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761448.638428:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761448.638433:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504051712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761448.638438:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761448.638447:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504051072:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761448.638476:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504051200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761448.638496:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761448.638506:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761448/real 1789761448] req@ffff9018d070ed80 x1876699504051200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761448.638514:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504051200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761448.638521:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761448.638523:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761448.638525:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504051200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761448.638529:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504051328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761448.638541:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761448.638547:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761448/real 1789761448] req@ffff9018d070df80 x1876699504051328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761448.638555:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504051328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761448.638560:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761448.638562:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761448.638564:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504051328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761448.638568:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504051456:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761448.638578:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761448.638584:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761448/real 1789761448] req@ffff9018d070ca80 x1876699504051456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761448.638591:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504051456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761448.638596:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761448.638598:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761448.638600:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504051456:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761448.638603:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504051584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761448.638613:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761448.638618:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761448/real 1789761448] req@ffff9018d070ce00 x1876699504051584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761448.638626:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504051584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761448.638631:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761448.638633:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761448.638636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504051584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761448.638639:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504051712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761448.638648:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761448.638653:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761448/real 1789761448] req@ffff9018d070d880 x1876699504051712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761448.638660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504051712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761448.638667:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761448.638670:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761448.638673:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504051712:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761449.278199:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761450.302187:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761451.326232:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761452.350159:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761453.374205:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761453.758179:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761453.758187:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761453.758190:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761453.758199:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1380 00000100:00080000:2.0:1789761453.758208:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761453.758214:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761453.758250:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504051840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761453.758280:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761453.758285:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761453.758287:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761453.758293:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1380 00000100:00080000:2.0:1789761453.758298:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761453.758303:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761453.758316:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761453.758331:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504051968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761453.758338:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504051840:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761453.758342:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761453.758355:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761453.758356:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761453.758360:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1380 00000100:00080000:2.0:1789761453.758362:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761453.758365:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761453.758369:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761453.758381:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504052096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761453.758393:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761453.758397:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761453.758400:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761453.758405:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1380 00000100:00080000:2.0:1789761453.758411:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761453.758416:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761453.758423:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761453.758436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504052224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761453.758445:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761453.758457:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761453.758460:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761453.758462:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761453.758466:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1380 00000100:00080000:2.0:1789761453.758471:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761453.758474:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761453.758478:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761453.758486:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504052352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761453.758493:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761453.758496:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761453.758497:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761453.758500:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1380 00000100:00080000:2.0:1789761453.758503:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761453.758509:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761453/real 1789761453] req@ffff9018d0f27800 x1876699504051840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761453.758514:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761453.758525:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761453.758546:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504051840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761453.758548:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504052480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761453.758580:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761453.758588:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761453.758592:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504051840:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761453.758621:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504051968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761453.758659:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761453.758671:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761453/real 1789761453] req@ffff9018d0f25180 x1876699504051968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761453.758680:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504051968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761453.758686:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761453.758688:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761453.758690:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504051968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761453.758695:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504052096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761453.758708:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761453.758715:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761453/real 1789761453] req@ffff9018d0f24a80 x1876699504052096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761453.758722:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504052096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761453.758727:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761453.758729:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761453.758731:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504052096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761453.758735:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504052224:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761453.758747:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761453.758754:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761453/real 1789761453] req@ffff9018d0f24380 x1876699504052224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761453.758761:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504052224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761453.758766:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761453.758768:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761453.758770:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504052224:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761453.758773:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504052352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761453.758784:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761453.758790:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761453/real 1789761453] req@ffff9018d0f25500 x1876699504052352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761453.758797:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504052352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761453.758802:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761453.758804:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761453.758805:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504052352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761453.758809:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504052480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761453.758819:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761453.758825:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761453/real 1789761453] req@ffff9018d0f24000 x1876699504052480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761453.758832:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504052480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761453.758838:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761453.758840:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761453.758842:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504052480:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761454.398158:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761455.422287:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761456.446276:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761457.470184:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761458.494227:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761458.878183:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761458.878194:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761458.878198:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761458.878209:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1385 00000100:00080000:2.0:1789761458.878221:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761458.878229:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761458.878279:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504052608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761458.878315:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761458.878318:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761458.878320:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761458.878324:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1385 00000100:00080000:2.0:1789761458.878327:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761458.878331:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761458.878341:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761458.878353:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504052736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761458.878362:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761458.878365:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761458.878368:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761458.878370:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504052608:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761458.878373:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1385 00000100:00080000:2.0:1789761458.878380:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761458.878382:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761458.878385:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761458.878393:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504052864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761458.878399:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761458.878402:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761458.878403:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761458.878407:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1385 00000100:00080000:2.0:1789761458.878411:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761458.878414:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761458.878418:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761458.878425:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504052992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761458.878432:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761458.878435:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761458.878437:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761458.878439:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1385 00000100:00080000:2.0:1789761458.878441:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761458.878444:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761458.878447:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761458.878451:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761458.878460:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504053120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761458.878468:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761458.878471:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761458.878473:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761458.878476:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1385 00000100:00080000:2.0:1789761458.878478:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761458.878480:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761458.878483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761458.878491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504053248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761458.878507:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761458/real 1789761458] req@ffff9018d0f27480 x1876699504052608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761458.878523:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504052608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761458.878533:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761458.878538:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761458.878542:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504052608:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761458.878568:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504052736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761458.878585:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761458.878593:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761458/real 1789761458] req@ffff9018d0f24e00 x1876699504052736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761458.878619:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504052736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761458.878625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761458.878628:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761458.878630:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504052736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761458.878635:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504052864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761458.878652:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761458.878659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761458/real 1789761458] req@ffff9018d0f26680 x1876699504052864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761458.878667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504052864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761458.878672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761458.878674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761458.878676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504052864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761458.878680:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504052992:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761458.878691:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761458.878697:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761458/real 1789761458] req@ffff9018d0f24700 x1876699504052992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761458.878704:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504052992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761458.878710:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761458.878711:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761458.878713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504052992:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761458.878717:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504053120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761458.878727:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761458.878733:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761458/real 1789761458] req@ffff9018d0f24000 x1876699504053120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761458.878740:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504053120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761458.878745:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761458.878747:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761458.878749:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504053120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761458.878752:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504053248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761458.878762:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761458.878768:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761458/real 1789761458] req@ffff9018d0f25500 x1876699504053248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761458.878776:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504053248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761458.878780:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761458.878782:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761458.878784:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504053248:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761459.518238:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761460.542209:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761461.566162:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761462.590197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761463.614172:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761463.998129:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761463.998138:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761463.998142:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761463.998150:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1390 00000100:00080000:0.0:1789761463.998160:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761463.998165:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761463.998195:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504053376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761463.998217:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761463.998221:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761463.998223:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761463.998228:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1390 00000100:00080000:0.0:1789761463.998231:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761463.998235:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761463.998245:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761463.998258:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504053504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761463.998267:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504053376:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761463.998268:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761463.998271:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761463.998273:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761463.998276:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1390 00000100:00080000:0.0:1789761463.998280:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761463.998283:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761463.998287:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761463.998296:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504053632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761463.998304:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761463.998308:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761463.998309:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761463.998312:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1390 00000100:00080000:0.0:1789761463.998315:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761463.998317:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761463.998322:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789761463.998327:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761463.998330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504053760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761463.998339:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761463.998342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761463.998344:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761463.998347:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1390 00000100:00080000:0.0:1789761463.998350:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761463.998353:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761463.998357:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761463.998364:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761463/real 1789761463] req@ffff9018c8b9a300 x1876699504053376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761463.998366:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504053888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761463.998376:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761463.998379:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761463.998381:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761463.998383:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504053376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761463.998384:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1390 00000100:00080000:0.0:1789761463.998388:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761463.998391:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761463.998395:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761463.998401:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789761463.998405:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504054016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761463.998405:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761463.998410:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504053376:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761463.998432:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504053504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761463.998457:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761463.998467:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761463/real 1789761463] req@ffff9018c8b9ad80 x1876699504053504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761463.998476:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504053504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761463.998482:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761463.998484:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761463.998486:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504053504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761463.998490:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504053632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761463.998503:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761463.998509:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761463/real 1789761463] req@ffff9018c8b99500 x1876699504053632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761463.998516:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504053632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761463.998522:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761463.998523:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761463.998525:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504053632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761463.998529:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504053760:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761463.998540:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761463.998546:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761463/real 1789761463] req@ffff9018c8b99c00 x1876699504053760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761463.998553:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504053760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761463.998571:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761463.998575:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761463.998578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504053760:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761463.998584:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504053888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761463.998605:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761463.998618:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761463/real 1789761463] req@ffff9018c8b98380 x1876699504053888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761463.998630:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504053888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761463.998636:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761463.998638:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761463.998640:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504053888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761463.998644:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504054016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761463.998658:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761463.998665:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761463/real 1789761463] req@ffff9018c8b9b100 x1876699504054016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761463.998674:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504054016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761463.998679:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761463.998681:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761463.998682:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504054016:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761464.638217:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761465.662199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761466.686224:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761467.710146:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761468.734219:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761469.118146:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761469.118156:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761469.118160:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761469.118170:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1395 00000100:00080000:3.0:1789761469.118183:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761469.118192:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761469.118248:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504054144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761469.118280:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761469.118284:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761469.118286:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761469.118292:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1395 00000100:00080000:3.0:1789761469.118296:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761469.118301:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761469.118314:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761469.118335:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504054144:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761469.118336:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504054272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761469.118354:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761469.118359:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761469.118361:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761469.118366:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1395 00000100:00080000:3.0:1789761469.118371:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761469.118375:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761469.118382:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761469.118396:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504054400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761469.118408:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761469.118412:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761469.118413:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761469.118418:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1395 00000100:00080000:3.0:1789761469.118423:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761469.118427:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761469.118431:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761469.118440:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761469.118454:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504054528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761469.118465:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761469.118470:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761469.118472:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761469.118476:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1395 00000100:00080000:3.0:1789761469.118481:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761469.118483:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761469/real 1789761469] req@ffff9018d070fb80 x1876699504054144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761469.118490:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761469.118501:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761469.118515:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504054144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761469.118518:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504054656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761469.118544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761469.118549:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761469.118551:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761469.118557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504054144:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761469.118557:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761469.118561:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761469.118568:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1395 00000100:00080000:3.0:1789761469.118572:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761469.118576:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761469.118581:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761469.118587:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504054272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761469.118595:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504054784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761469.118614:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761469.118628:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761469/real 1789761469] req@ffff9018d070c000 x1876699504054272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761469.118637:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504054272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761469.118647:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761469.118649:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761469.118652:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504054272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761469.118656:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504054400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761469.118668:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761469.118674:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761469/real 1789761469] req@ffff9018d070dc00 x1876699504054400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761469.118681:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504054400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761469.118687:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761469.118688:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761469.118690:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504054400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761469.118694:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504054528:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761469.118705:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761469.118711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761469/real 1789761469] req@ffff9018d070d180 x1876699504054528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761469.118719:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504054528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761469.118724:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761469.118725:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761469.118728:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504054528:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761469.118731:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504054656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761469.118741:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761469.118747:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761469/real 1789761469] req@ffff9018d070e300 x1876699504054656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761469.118754:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504054656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761469.118758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761469.118760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761469.118762:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504054656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761469.118774:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504054784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761469.118785:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761469.118791:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761469/real 1789761469] req@ffff9018d070e680 x1876699504054784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761469.118798:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504054784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761469.118803:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761469.118805:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761469.118807:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504054784:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761469.758143:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761470.782202:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761471.806145:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761472.830162:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761473.854175:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761474.238131:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761474.238141:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761474.238146:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761474.238156:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1400 00000100:00080000:0.0:1789761474.238167:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761474.238175:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761474.238207:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504054912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761474.238227:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761474.238230:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761474.238232:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761474.238234:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1400 00000100:00080000:0.0:1789761474.238237:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761474.238240:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761474.238247:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761474.238256:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504055040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761474.238264:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761474.238267:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789761474.238271:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504054912:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761474.238276:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761474.238281:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1400 00000100:00080000:0.0:1789761474.238283:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761474.238285:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761474.238289:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761474.238298:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504055168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761474.238305:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761474.238308:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761474.238309:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761474.238312:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1400 00000100:00080000:0.0:1789761474.238317:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761474.238319:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761474.238323:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789761474.238329:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761474.238333:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504055296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761474.238342:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761474.238345:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761474.238346:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761474.238348:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1400 00000100:00080000:0.0:1789761474.238351:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761474.238353:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761474.238356:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761474.238363:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504055424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761474.238370:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761474.238371:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761474/real 1789761474] req@ffff9018c8b9a680 x1876699504054912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761474.238375:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761474.238379:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761474.238388:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1400 00000100:00080000:0.0:1789761474.238392:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761474.238395:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761474.238399:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761474.238400:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504054912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761474.238409:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504055552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761474.238420:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761474.238428:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761474.238434:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504054912:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761474.238460:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504055040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761474.238501:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761474.238515:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761474/real 1789761474] req@ffff9018c8b98000 x1876699504055040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761474.238525:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504055040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761474.238530:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761474.238533:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761474.238535:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504055040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761474.238539:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504055168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761474.238554:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761474.238561:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761474/real 1789761474] req@ffff9018c8b9aa00 x1876699504055168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761474.238569:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504055168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761474.238579:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761474.238581:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761474.238583:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504055168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761474.238587:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504055296:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761474.238597:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761474.238603:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761474/real 1789761474] req@ffff9018c8b9b800 x1876699504055296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761474.238610:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504055296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761474.238615:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761474.238617:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761474.238619:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504055296:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761474.238622:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504055424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761474.238632:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761474.238638:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761474/real 1789761474] req@ffff9018c8b9b480 x1876699504055424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761474.238645:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504055424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761474.238650:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761474.238652:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761474.238654:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504055424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761474.238658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504055552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761474.238667:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761474.238672:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761474/real 1789761474] req@ffff9018c8b98e00 x1876699504055552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761474.238679:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504055552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761474.238684:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761474.238686:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761474.238688:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504055552:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761474.878158:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761475.902265:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761476.926193:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761477.950249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761478.974192:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761479.358130:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761479.358141:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761479.358145:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761479.358155:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1405 00000100:00080000:3.0:1789761479.358167:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761479.358184:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761479.358222:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504055680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761479.358253:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761479.358263:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761479.358266:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761479.358272:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1405 00000100:00080000:3.0:1789761479.358278:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761479.358283:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761479.358291:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761479.358299:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504055680:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761479.358306:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504055808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761479.358318:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761479.358321:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761479.358323:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761479.358326:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1405 00000100:00080000:3.0:1789761479.358330:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761479.358333:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761479.358336:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761479.358345:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504055936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761479.358352:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761479.358356:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761479.358357:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761479.358360:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1405 00000100:00080000:3.0:1789761479.358364:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761479.358365:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761479.358369:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761479.358374:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761479.358384:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504056064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761479.358433:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761479/real 1789761479] req@ffff9018d070d500 x1876699504055680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761479.358462:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504055680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761479.358494:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761479.358499:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761479.358503:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504055680:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761479.358525:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504055808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761479.358546:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761479.358555:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761479/real 1789761479] req@ffff9018d070f800 x1876699504055808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761479.358565:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504055808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761479.358572:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761479.358574:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761479.358577:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504055808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761479.358580:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504055936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761479.358594:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761479.358595:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761479.358599:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761479.358601:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761479.358603:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761479/real 1789761479] req@ffff9018d070f100 x1876699504055936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761479.358608:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1405 00000100:00080000:3.0:1789761479.358616:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761479.358624:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504055936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761479.358625:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761479.358634:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761479.358638:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761479.358645:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761479.358649:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504056192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761479.358653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504055936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761479.358672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504056064:192.168.203.158@tcp:8:kworker.0 00000100:00080000:3.0:1789761479.358695:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789761479.358702:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761479.358702:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761479.358705:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761479.358711:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1405 00000100:00080000:3.0:1789761479.358714:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761479.358717:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761479.358721:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761479.358725:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761479/real 1789761479] req@ffff9018d070e680 x1876699504056064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761479.358733:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504056320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761479.358749:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504056064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761479.358757:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761479.358760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761479.358763:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504056064:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761479.358778:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504056192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761479.358793:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761479.358800:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761479/real 1789761479] req@ffff9018d070e300 x1876699504056192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761479.358808:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504056192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761479.358813:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761479.358815:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761479.358817:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504056192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761479.358821:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504056320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761479.358830:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761479.358835:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761479/real 1789761479] req@ffff9018d070d180 x1876699504056320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761479.358843:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504056320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761479.358847:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761479.358849:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761479.358852:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504056320:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761479.998202:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761481.022221:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761482.046211:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761483.070182:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761484.094152:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761484.478105:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761484.478112:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761484.478116:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761484.478127:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1411 00000100:00080000:3.0:1789761484.478137:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761484.478143:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761484.478181:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504056448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761484.478214:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761484.478218:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761484.478219:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761484.478224:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1411 00000100:00080000:3.0:1789761484.478227:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761484.478232:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761484.478242:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761484.478260:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504056448:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761484.478260:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504056576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761484.478279:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761484.478283:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761484.478285:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761484.478290:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1411 00000100:00080000:3.0:1789761484.478294:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761484.478299:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761484.478304:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761484.478318:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504056704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761484.478331:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761484.478331:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761484.478338:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761484.478341:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761484.478347:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1411 00000100:00080000:3.0:1789761484.478354:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761484.478358:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761484.478364:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761484.478376:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761484/real 1789761484] req@ffff9018d070dc00 x1876699504056448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761484.478379:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504056832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761484.478423:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761484.478428:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789761484.478430:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504056448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761484.478432:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761484.478442:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1411 00000100:00080000:1.0:1789761484.478445:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761484.478449:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761484.478452:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761484.478457:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504056448:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761484.478460:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761484.478469:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761484.478481:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504056576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761484.478485:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504056960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761484.478500:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761484.478504:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1789761484.478508:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761484.478509:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761484.478515:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1411 00000100:00080000:3.0:1789761484.478520:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761484.478524:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761484/real 1789761484] req@ffff9018d070c000 x1876699504056576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761484.478527:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761484.478534:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761484.478541:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504056576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761484.478546:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504057088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761484.478555:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761484.478562:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761484.478567:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504056576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761484.478571:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504056704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761484.478586:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761484.478594:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761484/real 1789761484] req@ffff9018d070fb80 x1876699504056704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761484.478601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504056704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761484.478607:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761484.478609:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761484.478612:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504056704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761484.478615:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504056832:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761484.478626:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761484.478632:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761484/real 1789761484] req@ffff9018d070d880 x1876699504056832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761484.478639:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504056832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761484.478644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761484.478646:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761484.478648:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504056832:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761484.478658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504056960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761484.478668:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761484.478674:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761484/real 1789761484] req@ffff9018d070ce00 x1876699504056960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761484.478680:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504056960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761484.478686:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761484.478688:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761484.478691:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504056960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761484.478694:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504057088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761484.478703:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761484.478709:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761484/real 1789761484] req@ffff9018d070ca80 x1876699504057088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761484.478716:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504057088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761484.478722:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761484.478723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761484.478725:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504057088:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761485.118214:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761486.142187:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761487.166164:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761488.190175:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761489.214172:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761489.598157:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761489.598168:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761489.598173:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761489.598183:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1416 00000100:00080000:3.0:1789761489.598201:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761489.598210:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761489.598253:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504057216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761489.598275:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761489.598278:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761489.598280:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761489.598283:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1416 00000100:00080000:3.0:1789761489.598286:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761489.598288:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761489.598296:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761489.598304:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504057344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761489.598316:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761489.598318:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761489.598320:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761489.598322:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1416 00000100:00080000:3.0:1789761489.598324:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761489.598326:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761489.598331:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761489.598332:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504057216:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761489.598339:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504057472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761489.598347:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761489.598350:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761489.598351:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761489.598353:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1416 00000100:00080000:3.0:1789761489.598356:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761489.598358:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761489.598361:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761489.598383:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504057600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761489.598390:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761489.598393:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761489.598394:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761489.598397:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1416 00000100:00080000:3.0:1789761489.598400:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761489.598403:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761489.598406:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761489.598414:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504057728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761489.598422:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761489.598423:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761489.598428:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761489.598429:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761489.598433:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1416 00000100:00080000:3.0:1789761489.598436:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761489.598438:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761489.598441:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761489.598448:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504057856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761489.598473:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761489/real 1789761489] req@ffff9018d070df80 x1876699504057216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761489.598494:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504057216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761489.598506:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761489.598510:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761489.598515:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504057216:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761489.598524:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504057344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761489.598550:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761489.598566:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761489/real 1789761489] req@ffff9018d070ed80 x1876699504057344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761489.598580:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504057344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761489.598589:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761489.598593:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761489.598597:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504057344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761489.598622:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504057472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761489.598647:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761489.598660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761489/real 1789761489] req@ffff9018d070f480 x1876699504057472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761489.598673:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504057472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761489.598682:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761489.598686:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761489.598690:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504057472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761489.598696:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504057600:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761489.598730:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761489.598745:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761489/real 1789761489] req@ffff9018d070ca80 x1876699504057600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761489.598758:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504057600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761489.598767:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761489.598771:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761489.598775:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504057600:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761489.598781:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504057728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761489.598804:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761489.598817:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761489/real 1789761489] req@ffff9018d070ce00 x1876699504057728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761489.598831:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504057728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761489.598837:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761489.598851:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761489.598854:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504057728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761489.598858:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504057856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761489.598873:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761489.598881:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761489/real 1789761489] req@ffff9018d070d880 x1876699504057856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761489.598888:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504057856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761489.598894:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761489.598896:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761489.598898:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504057856:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761490.238183:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761491.262176:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761492.286227:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761493.310158:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761494.334155:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761494.718118:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761494.718127:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761494.718130:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761494.718138:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1421 00000100:00080000:2.0:1789761494.718147:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761494.718152:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761494.718183:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504057984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761494.718205:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761494.718209:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761494.718211:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761494.718216:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1421 00000100:00080000:2.0:1789761494.718219:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761494.718224:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761494.718235:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761494.718249:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504058112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761494.718256:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504057984:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761494.718261:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761494.718266:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761494.718268:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761494.718271:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1421 00000100:00080000:2.0:1789761494.718274:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761494.718276:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761494.718280:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761494.718288:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504058240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761494.718294:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761494.718297:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761494.718298:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761494.718301:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1421 00000100:00080000:2.0:1789761494.718303:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761494.718305:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761494.718309:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761494.718315:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504058368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761494.718329:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761494.718343:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761494.718345:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761494.718348:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1421 00000100:00080000:2.0:1789761494.718351:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761494.718354:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761494.718357:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761494.718369:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504058496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761494.718382:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504058112:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761494.718382:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761494.718391:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761494.718395:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761494.718400:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1421 00000100:00080000:2.0:1789761494.718405:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761494.718410:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761494.718416:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761494.718425:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504058240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761494.718431:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504058624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761494.718467:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504058368:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761494.718507:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504058496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761494.718530:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504058624:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761495.358168:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761496.382156:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761497.406207:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761497.790385:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:3.0:1789761497.790447:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761497.790469:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761497.790524:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761497.790536:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761497.790547:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761497.790559:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761497.790571:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789761497.790578:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761494/real 1789761497] req@ffff9018d0f24a80 x1876699504058112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761497.790610:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504058112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761497.790625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761497.790639:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761497.790644:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504058112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761497.790653:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761494/real 1789761497] req@ffff9018d0f25180 x1876699504058240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761497.790673:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504058240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761497.790682:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761497.790686:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761497.790689:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504058240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761497.790698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761494/real 1789761497] req@ffff9018d0f27800 x1876699504058368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761497.790712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504058368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761497.790720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761497.790725:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761497.790728:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504058368:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761497.790736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761494/real 1789761497] req@ffff9018d0f24380 x1876699504057984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761497.790750:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504057984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761497.790760:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761497.790764:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761497.790768:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504057984:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761497.790800:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761494/real 1789761497] req@ffff9018d0f26d80 x1876699504058496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761497.790816:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504058496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761497.790825:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761497.790829:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761497.790832:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504058496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761497.790840:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761494/real 1789761497] req@ffff9018d0f27100 x1876699504058624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761497.790852:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504058624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761497.790861:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761497.790865:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761497.790869:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504058624:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761498.430235:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761499.454249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761499.838116:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761499.838127:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761499.838132:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761499.838143:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1426 00000100:00080000:2.0:1789761499.838155:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761499.838163:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761499.838201:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504058752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761499.838232:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761499.838237:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761499.838239:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761499.838243:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1426 00000100:00080000:2.0:1789761499.838247:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761499.838251:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761499.838264:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761499.838280:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504058880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761499.838291:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761499.838311:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789761499.838313:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504058752:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761499.838313:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761499.838323:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1426 00000100:00080000:2.0:1789761499.838327:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761499.838332:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761499.838338:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761499.838355:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504059008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761499.838366:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761499.838370:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761499.838372:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761499.838376:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1426 00000100:00080000:2.0:1789761499.838390:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761499.838398:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789761499.838401:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789761499.838407:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761499.838422:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504059136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761499.838433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761499.838437:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761499.838439:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761499.838443:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1426 00000100:00080000:2.0:1789761499.838447:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761499.838454:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761499/real 1789761499] req@ffff9018d0f26300 x1876699504058752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761499.838454:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761499.838462:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761499.838481:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504059264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761499.838486:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504058752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761499.838505:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761499.838509:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761499.838514:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789761499.838516:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761499.838521:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761499.838525:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504058752:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761499.838529:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1426 00000100:00080000:2.0:1789761499.838536:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761499.838541:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761499.838547:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761499.838559:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504059392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761499.838567:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504058880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761499.838603:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761499.838619:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761499/real 1789761499] req@ffff9018d0f25f80 x1876699504058880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761499.838633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504058880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761499.838642:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761499.838646:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761499.838649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504058880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761499.838656:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504059008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761499.838680:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761499.838692:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761499/real 1789761499] req@ffff9018d0f25c00 x1876699504059008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761499.838705:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504059008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761499.838727:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761499.838731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761499.838734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504059008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761499.838741:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504059136:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761499.838763:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761499.838778:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761499/real 1789761499] req@ffff9018d0f26a00 x1876699504059136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761499.838789:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504059136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761499.838796:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761499.838799:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761499.838802:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504059136:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761499.838807:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504059264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761499.838827:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761499.838838:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761499/real 1789761499] req@ffff9018d0f27100 x1876699504059264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761499.838849:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504059264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761499.838857:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761499.838860:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761499.838863:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504059264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761499.838892:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504059392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761499.838913:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761499.838926:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761499/real 1789761499] req@ffff9018d0f26d80 x1876699504059392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761499.838938:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504059392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761499.838948:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761499.838951:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761499.838955:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504059392:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761500.478206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761501.502223:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761502.526195:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761503.550235:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761504.574194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761504.958171:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761504.958183:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761504.958188:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761504.958202:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1431 00000100:00080000:2.0:1789761504.958215:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761504.958225:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761504.958284:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504059520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761504.958310:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761504.958313:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761504.958314:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761504.958318:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1431 00000100:00080000:2.0:1789761504.958321:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761504.958324:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761504.958334:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761504.958348:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504059648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761504.958358:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504059520:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761504.958362:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761504.958368:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761504.958370:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761504.958375:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1431 00000100:00080000:2.0:1789761504.958380:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761504.958385:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761504.958391:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761504.958405:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504059776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761504.958416:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761504.958420:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761504.958423:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761504.958427:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1431 00000100:00080000:2.0:1789761504.958432:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761504.958436:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761504.958441:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761504.958449:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761504.958463:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504059904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761504.958472:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761504.958476:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761504.958478:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761504.958482:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1431 00000100:00080000:2.0:1789761504.958487:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761504.958494:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761504/real 1789761504] req@ffff9018d0f24380 x1876699504059520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761504.958494:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761504.958505:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761504.958516:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504059520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761504.958525:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504060032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761504.958528:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761504.958538:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789761504.958541:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761504.958545:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504059520:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761504.958547:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761504.958552:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761504.958556:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1431 00000100:00080000:2.0:1789761504.958559:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761504.958562:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761504.958566:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761504.958574:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504060160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761504.958577:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504059648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761504.958604:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761504.958615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761504/real 1789761504] req@ffff9018d0f27800 x1876699504059648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761504.958624:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504059648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761504.958629:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761504.958632:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761504.958634:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504059648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761504.958639:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504059776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761504.958650:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761504.958656:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761504/real 1789761504] req@ffff9018d0f25180 x1876699504059776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761504.958668:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504059776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761504.958673:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761504.958675:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761504.958678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504059776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761504.958682:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504059904:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761504.958694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761504.958701:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761504/real 1789761504] req@ffff9018d0f24a80 x1876699504059904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761504.958708:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504059904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761504.958713:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761504.958715:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761504.958717:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504059904:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761504.958720:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504060032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761504.958731:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761504.958737:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761504/real 1789761504] req@ffff9018d0f25500 x1876699504060032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761504.958744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504060032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761504.958749:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761504.958751:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761504.958753:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504060032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761504.958765:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504060160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761504.958775:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761504.958781:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761504/real 1789761504] req@ffff9018d0f24000 x1876699504060160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761504.958788:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504060160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761504.958793:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761504.958795:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761504.958797:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504060160:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761505.598200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761506.622191:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761507.646266:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761508.670173:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761509.694189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761510.078178:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761510.078189:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761510.078194:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761510.078205:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1436 00000100:00080000:0.0:1789761510.078229:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761510.078239:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761510.078294:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504060288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761510.078327:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761510.078332:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761510.078335:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761510.078339:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1436 00000100:00080000:0.0:1789761510.078344:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761510.078348:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761510.078358:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761510.078371:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504060416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761510.078380:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761510.078387:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761510.078390:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761510.078391:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504060288:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761510.078398:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1436 00000100:00080000:0.0:1789761510.078405:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761510.078411:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761510.078414:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761510.078427:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504060544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761510.078435:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761510.078437:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761510.078439:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761510.078443:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1436 00000100:00080000:0.0:1789761510.078457:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761510.078461:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761510.078465:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789761510.078470:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761510.078479:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504060672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761510.078491:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761510.078494:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761510.078495:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761510.078499:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1436 00000100:00080000:0.0:1789761510.078502:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761510.078505:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761510.078509:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761510.078515:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504060800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761510.078522:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761510.078526:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761510/real 1789761510] req@ffff9018c8b99880 x1876699504060288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761510.078527:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761510.078532:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761510.078542:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1436 00000100:00080000:0.0:1789761510.078546:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761510.078555:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761510.078562:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761510.078572:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504060928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761510.078582:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504060288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761510.078604:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761510.078616:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761510.078619:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504060288:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761510.078642:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504060416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761510.078663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761510.078673:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761510/real 1789761510] req@ffff9018c8b99f80 x1876699504060416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761510.078682:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504060416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761510.078688:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761510.078690:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761510.078692:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504060416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761510.078697:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504060544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761510.078707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761510.078713:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761510/real 1789761510] req@ffff9018c8b9bb80 x1876699504060544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761510.078721:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504060544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761510.078726:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761510.078728:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761510.078730:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504060544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761510.078735:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504060672:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761510.078746:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761510.078752:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761510/real 1789761510] req@ffff9018c8b98700 x1876699504060672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761510.078759:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504060672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761510.078765:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761510.078767:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761510.078768:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504060672:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761510.078772:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504060800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761510.078781:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761510.078787:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761510/real 1789761510] req@ffff9018c8b98a80 x1876699504060800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761510.078794:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504060800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761510.078799:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761510.078802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761510.078803:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504060800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761510.078807:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504060928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761510.078820:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761510.078830:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761510/real 1789761510] req@ffff9018c8b98e00 x1876699504060928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761510.078843:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504060928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761510.078852:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761510.078855:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761510.078858:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504060928:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761510.718183:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761511.742180:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761512.766169:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761513.790223:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761514.814221:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761515.198155:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761515.198166:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761515.198171:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761515.198195:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1441 00000100:00080000:3.0:1789761515.198206:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761515.198214:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761515.198250:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504061056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761515.198281:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761515.198286:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761515.198288:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761515.198295:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1441 00000100:00080000:3.0:1789761515.198301:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761515.198306:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761515.198311:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504061056:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789761515.198319:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761515.198337:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504061184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761515.198350:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761515.198355:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761515.198358:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761515.198364:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1441 00000800:00000100:2.0:1789761515.198367:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761515.198372:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761515.198379:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761515.198385:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761515.198399:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504061312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761515.198402:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761515/real 1789761515] req@ffff9018d070fb80 x1876699504061056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761515.198423:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504061056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761515.198434:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761515.198438:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761515.198443:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504061056:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761515.198470:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504061184:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761515.198477:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761515.198483:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761515.198485:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761515.198492:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1441 00000100:00080000:3.0:1789761515.198497:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789761515.198500:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761515.198515:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761515/real 1789761515] req@ffff9018d070c000 x1876699504061184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761515.198534:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504061184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761515.198540:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761515.198543:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761515.198545:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504061184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761515.198555:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504061312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761515.198571:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761515.198581:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761515/real 1789761515] req@ffff9018d070dc00 x1876699504061312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761515.198590:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504061312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761515.198595:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761515.198598:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761515.198600:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504061312:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761515.198612:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761515.198619:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761515.198687:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504061440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761515.198714:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761515.198718:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761515.198721:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761515.198725:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1441 00000100:00080000:3.0:1789761515.198730:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761515.198735:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761515.198742:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761515.198748:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504061440:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761515.198756:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504061568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761515.198770:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761515.198774:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1789761515.198778:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761515.198779:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761515.198786:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1441 00000100:00080000:3.0:1789761515.198791:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761515.198794:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761515/real 1789761515] req@ffff9018d070d180 x1876699504061440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761515.198799:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761515.198810:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761515.198813:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504061440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761515.198824:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504061696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761515.198829:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761515.198837:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761515.198843:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504061440:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761515.198856:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504061568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761515.198871:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761515.198879:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761515/real 1789761515] req@ffff9018d070e300 x1876699504061568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761515.198886:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504061568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761515.198891:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761515.198893:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761515.198895:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504061568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761515.198899:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504061696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761515.198909:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761515.198915:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761515/real 1789761515] req@ffff9018d070e680 x1876699504061696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761515.198922:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504061696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761515.198927:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761515.198928:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761515.198931:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504061696:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761515.838155:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761516.862231:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761517.886223:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761518.910217:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761519.934183:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761520.318123:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761520.318131:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761520.318134:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761520.318155:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1446 00000100:00080000:0.0:1789761520.318166:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761520.318173:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761520.318210:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504061824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761520.318232:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761520.318236:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761520.318238:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761520.318243:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1446 00000100:00080000:0.0:1789761520.318247:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761520.318251:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761520.318261:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761520.318266:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504061824:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761520.318274:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504061952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761520.318284:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761520.318287:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761520.318289:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761520.318292:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1446 00000100:00080000:0.0:1789761520.318295:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761520.318299:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761520.318303:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761520.318312:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504062080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761520.318320:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789761520.318322:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761520.318324:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761520.318325:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761520.318329:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1446 00000100:00080000:0.0:1789761520.318331:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761520.318335:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761520.318339:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761520.318349:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504062208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761520.318359:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761520.318362:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789761520.318362:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761520/real 1789761520] req@ffff9018c8b9b480 x1876699504061824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761520.318364:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761520.318368:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1446 00000100:00080000:0.0:1789761520.318371:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761520.318375:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761520.318379:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761520.318380:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504061824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761520.318390:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504062336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761520.318391:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761520.318395:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761520.318399:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761520.318399:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504061824:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761520.318403:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761520.318405:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761520.318408:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1446 00000100:00080000:0.0:1789761520.318411:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761520.318414:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761520.318418:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761520.318423:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504061952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761520.318426:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504062464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761520.318456:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761520.318471:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761520/real 1789761520] req@ffff9018c8b9b800 x1876699504061952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761520.318481:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504061952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761520.318487:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761520.318489:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761520.318491:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504061952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761520.318495:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504062080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761520.318512:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761520.318519:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761520/real 1789761520] req@ffff9018c8b9aa00 x1876699504062080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761520.318526:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504062080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761520.318531:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761520.318533:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761520.318535:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504062080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761520.318538:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504062208:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761520.318549:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761520.318555:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761520/real 1789761520] req@ffff9018c8b98000 x1876699504062208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761520.318562:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504062208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761520.318567:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761520.318569:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761520.318571:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504062208:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761520.318574:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504062336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761520.318583:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761520.318589:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761520/real 1789761520] req@ffff9018c8b9a680 x1876699504062336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761520.318601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504062336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761520.318605:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761520.318607:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761520.318609:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504062336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761520.318629:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504062464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761520.318641:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761520.318647:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761520/real 1789761520] req@ffff9018c8b9b100 x1876699504062464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761520.318654:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504062464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761520.318659:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761520.318661:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761520.318663:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504062464:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761520.958236:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761521.982208:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761523.006242:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761524.030182:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761525.054210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761525.438129:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761525.438139:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761525.438143:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761525.438153:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1451 00000100:00080000:0.0:1789761525.438164:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761525.438172:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761525.438208:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504062592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761525.438237:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761525.438241:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761525.438243:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761525.438248:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1451 00000100:00080000:0.0:1789761525.438252:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761525.438257:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761525.438267:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761525.438283:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504062720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761525.438288:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504062592:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761525.438312:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761525.438316:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761525.438319:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761525.438324:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1451 00000100:00080000:0.0:1789761525.438329:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761525.438333:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761525.438340:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761525.438345:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761525.438357:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504062848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761525.438370:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761525.438374:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761525.438376:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761525.438380:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1451 00000100:00080000:0.0:1789761525.438385:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761525.438387:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761525/real 1789761525] req@ffff9018c8b98380 x1876699504062592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761525.438393:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761525.438404:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761525.438415:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504062592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761525.438421:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504062976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761525.438436:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761525.438443:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761525.438443:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761525.438450:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504062592:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761525.438451:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761525.438457:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761525.438463:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1451 00000100:00080000:0.0:1789761525.438468:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761525.438472:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504062720:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761525.438474:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761525.438484:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761525.438498:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761525.438502:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504063104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761525.438517:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761525/real 1789761525] req@ffff9018c8b99c00 x1876699504062720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761525.438518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761525.438527:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761525.438532:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761525.438537:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504062720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761525.438542:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1451 00000100:00080000:2.0:1789761525.438548:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761525.438551:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761525.438553:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761525.438558:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504062720:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761525.438562:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761525.438567:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504062848:192.168.203.158@tcp:38:kworker.0 00000008:00100000:0.0:1789761525.438572:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761525.438583:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761525.438592:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504063232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761525.438593:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761525/real 1789761525] req@ffff9018c8b99500 x1876699504062848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761525.438625:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504062848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761525.438632:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761525.438635:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761525.438637:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504062848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761525.438642:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504062976:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761525.438654:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761525.438660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761525/real 1789761525] req@ffff9018c8b9ad80 x1876699504062976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761525.438667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504062976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761525.438672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761525.438674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761525.438676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504062976:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761525.438686:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504063104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761525.438697:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761525.438702:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761525/real 1789761525] req@ffff9018c8b9a300 x1876699504063104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761525.438709:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504063104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761525.438714:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761525.438716:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761525.438718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504063104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761525.438721:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504063232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761525.438731:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761525.438736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761525/real 1789761525] req@ffff9018c8b98380 x1876699504063232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761525.438744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504063232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761525.438749:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761525.438750:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761525.438752:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504063232:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761526.078171:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761527.102191:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761528.126216:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761529.150219:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761530.174189:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761530.558176:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761530.558185:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761530.558189:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761530.558199:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1457 00000100:00080000:3.0:1789761530.558209:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761530.558214:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761530.558244:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504063360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761530.558266:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761530.558268:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761530.558271:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761530.558274:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1457 00000100:00080000:3.0:1789761530.558276:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761530.558280:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761530.558290:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761530.558304:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504063488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761530.558311:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504063360:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761530.558322:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761530.558326:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761530.558329:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761530.558334:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1457 00000100:00080000:3.0:1789761530.558340:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761530.558345:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761530.558351:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761530.558366:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761530.558368:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504063616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761530.558382:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761530.558393:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761530.558396:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761530.558401:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761530/real 1789761530] req@ffff9018d070f100 x1876699504063360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761530.558402:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1457 00000100:00080000:3.0:1789761530.558411:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761530.558418:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761530.558421:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504063360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789761530.558428:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1789761530.558431:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761530.558436:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761530.558439:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504063360:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761530.558447:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504063744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761530.558462:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504063488:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761530.558463:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761530.558472:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761530.558475:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789761530.558490:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761530.558492:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1457 00000100:00080000:3.0:1789761530.558502:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761530.558503:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761530/real 1789761530] req@ffff9018d070f800 x1876699504063488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761530.558512:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761530.558523:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761530.558530:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504063488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761530.558541:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504063872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761530.558544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761530.558553:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761530.558562:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504063488:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761530.558563:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761530.558571:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789761530.558576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504063616:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761530.558577:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761530.558588:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1457 00000100:00080000:3.0:1789761530.558594:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761530.558599:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789761530.558605:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789761530.558606:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761530.558616:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761530/real 1789761530] req@ffff9018d070d500 x1876699504063616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761530.558625:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504064000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761530.558630:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504063616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761530.558652:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761530.558657:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761530.558669:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504063616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761530.558684:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504063744:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761530.558707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761530.558719:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761530/real 1789761530] req@ffff9018d070e680 x1876699504063744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761530.558732:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504063744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761530.558741:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761530.558745:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761530.558749:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504063744:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761530.558755:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504063872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761530.558770:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761530.558778:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761530/real 1789761530] req@ffff9018d070e300 x1876699504063872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761530.558786:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504063872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761530.558791:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761530.558793:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761530.558795:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504063872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761530.558798:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504064000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761530.558808:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761530.558813:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761530/real 1789761530] req@ffff9018d070d180 x1876699504064000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761530.558820:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504064000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761530.558825:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761530.558827:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761530.558829:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504064000:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761531.198258:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761532.222158:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761533.246255:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761534.270200:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761535.294249:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761535.678081:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761535.678089:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761535.678092:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761535.678100:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1462 00000100:00080000:3.0:1789761535.678109:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761535.678115:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761535.678145:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504064128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761535.678164:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761535.678167:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761535.678168:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761535.678171:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1462 00000100:00080000:3.0:1789761535.678173:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761535.678176:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761535.678184:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761535.678192:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504064256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761535.678200:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761535.678204:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504064128:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761535.678205:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761535.678210:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761535.678216:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1462 00000100:00080000:3.0:1789761535.678220:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761535.678224:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761535.678230:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761535.678249:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504064384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761535.678261:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761535.678273:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761535.678278:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761535.678280:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761535.678284:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1462 00000100:00080000:3.0:1789761535.678288:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761535.678293:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761535/real 1789761535] req@ffff9018d070dc00 x1876699504064128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761535.678293:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761535.678302:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761535.678316:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504064512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761535.678325:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761535.678327:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504064128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761535.678328:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761535.678331:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761535.678338:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1462 00000100:00080000:3.0:1789761535.678343:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761535.678348:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761535.678354:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789761535.678357:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761535.678362:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504064640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761535.678367:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789761535.678377:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761535.678378:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504064128:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761535.678380:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761535.678384:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761535.678390:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1462 00000100:00080000:3.0:1789761535.678393:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761535.678395:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761535.678398:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761535.678404:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504064768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761535.678413:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504064256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761535.678436:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761535.678446:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761535/real 1789761535] req@ffff9018d070c000 x1876699504064256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761535.678455:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504064256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761535.678461:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761535.678463:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761535.678465:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504064256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761535.678469:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504064384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761535.678479:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761535.678485:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761535/real 1789761535] req@ffff9018d070fb80 x1876699504064384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761535.678498:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504064384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761535.678505:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761535.678509:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761535.678512:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504064384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761535.678518:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504064512:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761535.678538:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761535.678550:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761535/real 1789761535] req@ffff9018d070d880 x1876699504064512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761535.678563:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504064512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761535.678572:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761535.678574:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761535.678578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504064512:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761535.678584:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504064640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761535.678599:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761535.678606:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761535/real 1789761535] req@ffff9018d070ce00 x1876699504064640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761535.678617:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504064640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761535.678631:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761535.678635:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761535.678638:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504064640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761535.678656:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504064768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761535.678677:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761535.678689:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761535/real 1789761535] req@ffff9018d070ca80 x1876699504064768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761535.678702:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504064768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761535.678711:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761535.678714:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761535.678718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504064768:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761536.318167:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761537.342179:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761538.366187:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761539.390166:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761540.414195:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761540.798108:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761540.798117:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761540.798121:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761540.798130:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1467 00000100:00080000:0.0:1789761540.798138:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761540.798143:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761540.798174:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504064896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761540.798194:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761540.798197:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761540.798198:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761540.798201:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1467 00000100:00080000:0.0:1789761540.798203:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761540.798205:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761540.798213:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761540.798222:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504065024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761540.798229:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761540.798232:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761540.798233:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761540.798236:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1467 00000100:00080000:0.0:1789761540.798238:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761540.798243:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761540.798244:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504064896:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789761540.798246:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761540.798256:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504065152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761540.798266:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761540.798270:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761540.798272:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761540.798277:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1467 00000100:00080000:0.0:1789761540.798282:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761540.798287:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761540.798292:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761540.798310:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504065280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761540.798310:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761540.798325:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761540.798329:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761540.798331:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761540.798340:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1467 00000100:00080000:0.0:1789761540.798346:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761540.798351:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761540/real 1789761540] req@ffff9018c8b9b100 x1876699504064896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761540.798353:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761540.798363:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761540.798371:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504064896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761540.798380:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504065408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761540.798382:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761540.798391:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761540.798398:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504064896:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761540.798404:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761540.798408:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504065024:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761540.798413:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761540.798417:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761540.798422:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1467 00000100:00080000:0.0:1789761540.798427:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761540.798430:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761540.798435:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761540.798436:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761540.798450:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504065536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761540.798457:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761540/real 1789761540] req@ffff9018c8b9a680 x1876699504065024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761540.798475:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504065024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761540.798485:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761540.798490:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761540.798493:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504065024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761540.798517:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504065152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761540.798537:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761540.798547:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761540/real 1789761540] req@ffff9018c8b98000 x1876699504065152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761540.798560:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504065152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761540.798568:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761540.798572:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761540.798575:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504065152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761540.798581:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504065280:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761540.798597:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761540.798603:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761540/real 1789761540] req@ffff9018c8b9aa00 x1876699504065280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761540.798611:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504065280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761540.798617:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761540.798618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761540.798620:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504065280:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761540.798624:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504065408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761540.798637:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761540.798648:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761540/real 1789761540] req@ffff9018c8b9b800 x1876699504065408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761540.798660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504065408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761540.798669:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761540.798673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761540.798676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504065408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761540.798682:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504065536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761540.798701:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761540.798711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761540/real 1789761540] req@ffff9018c8b9b480 x1876699504065536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761540.798730:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504065536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761540.798738:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761540.798741:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761540.798745:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504065536:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761541.438161:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761542.462187:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761543.486154:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761544.510197:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761545.534189:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761545.918180:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761545.918188:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761545.918192:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761545.918200:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1472 00000100:00080000:3.0:1789761545.918209:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761545.918215:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761545.918244:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504065664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761545.918269:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761545.918272:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761545.918273:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761545.918276:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1472 00000100:00080000:3.0:1789761545.918278:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761545.918280:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761545.918287:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761545.918295:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504065792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761545.918303:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761545.918305:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761545.918307:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761545.918309:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1472 00000100:00080000:3.0:1789761545.918311:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761545.918314:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761545.918317:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761545.918334:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504065920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761545.918349:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761545.918352:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761545.918355:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761545.918361:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1472 00000100:00100000:2.0:1789761545.918367:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504065664:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761545.918367:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761545.918382:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761545.918390:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761545.918411:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504066048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761545.918420:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761545.918422:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761545.918423:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761545.918426:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1472 00000100:00080000:3.0:1789761545.918453:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761545.918458:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761545.918464:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761545.918469:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761545.918487:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504066176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761545.918502:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761545.918506:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761545.918509:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761545.918514:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1472 00000100:00080000:3.0:1789761545.918519:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761545.918523:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761545/real 1789761545] req@ffff9018d070f480 x1876699504065664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761545.918530:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761545.918543:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761545.918550:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504065664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761545.918558:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504066304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761545.918579:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761545.918586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761545.918594:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504065664:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761545.918602:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504065792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761545.918623:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761545.918633:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761545/real 1789761545] req@ffff9018d070ed80 x1876699504065792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761545.918642:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504065792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761545.918648:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761545.918651:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761545.918653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504065792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761545.918678:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504065920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761545.918690:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761545.918696:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761545/real 1789761545] req@ffff9018d070df80 x1876699504065920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761545.918704:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504065920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761545.918710:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761545.918712:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761545.918714:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504065920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761545.918719:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504066048:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761545.918730:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761545.918737:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761545/real 1789761545] req@ffff9018d070ca80 x1876699504066048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761545.918744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504066048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761545.918750:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761545.918752:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761545.918754:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504066048:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761545.918758:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504066176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761545.918776:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761545.918787:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761545/real 1789761545] req@ffff9018d070ce00 x1876699504066176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761545.918798:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504066176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761545.918806:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761545.918810:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761545.918813:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504066176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761545.918830:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504066304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761545.918850:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761545.918861:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761545/real 1789761545] req@ffff9018d070d880 x1876699504066304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761545.918874:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504066304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761545.918882:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761545.918897:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761545.918900:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504066304:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761546.558207:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761547.582157:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761548.606197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761549.630186:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761550.654200:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761551.038140:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761551.038150:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761551.038154:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761551.038164:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1477 00000100:00080000:0.0:1789761551.038175:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761551.038182:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761551.038216:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504066432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761551.038249:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761551.038253:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761551.038255:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761551.038260:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1477 00000100:00080000:0.0:1789761551.038265:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761551.038271:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761551.038280:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761551.038287:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504066432:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761551.038299:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504066560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761551.038311:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761551.038315:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761551.038318:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761551.038322:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1477 00000100:00080000:0.0:1789761551.038327:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761551.038330:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761551.038335:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761551.038350:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504066688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761551.038350:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761551.038362:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761551.038366:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761551.038368:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761551.038372:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1477 00000100:00080000:0.0:1789761551.038376:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761551.038380:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761551.038385:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761551.038398:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504066816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761551.038408:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761551.038412:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761551.038413:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761551.038417:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1477 00000100:00080000:0.0:1789761551.038420:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761551.038423:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761551.038427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761551.038437:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504066944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761551.038446:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761551.038449:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761551.038450:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761551.038454:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1477 00000100:00080000:0.0:1789761551.038457:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761551.038460:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761551.038464:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761551.038473:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504067072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761551.038479:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761551/real 1789761551] req@ffff9018c8b98e00 x1876699504066432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761551.038504:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504066432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761551.038518:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761551.038523:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761551.038528:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504066432:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761551.038660:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504066560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761551.038743:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761551.038759:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761551/real 1789761551] req@ffff9018c8b98a80 x1876699504066560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761551.038773:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504066560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761551.038783:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761551.038786:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761551.038790:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504066560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761551.038797:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504066688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761551.038815:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761551.038825:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761551/real 1789761551] req@ffff9018c8b98700 x1876699504066688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761551.038838:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504066688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761551.038844:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761551.038846:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761551.038848:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504066688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761551.038852:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504066816:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761551.038866:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761551.038873:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761551/real 1789761551] req@ffff9018c8b9bb80 x1876699504066816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761551.038881:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504066816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761551.038886:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761551.038888:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761551.038890:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504066816:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761551.038893:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504066944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761551.038902:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761551.038908:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761551/real 1789761551] req@ffff9018c8b99f80 x1876699504066944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761551.038933:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504066944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761551.038938:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761551.038940:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761551.038942:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504066944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761551.038947:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504067072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761551.038962:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761551.038973:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761551/real 1789761551] req@ffff9018c8b99880 x1876699504067072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761551.038993:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504067072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761551.039001:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761551.039004:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761551.039008:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504067072:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761551.678146:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761552.702208:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761553.726179:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761554.750209:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761555.774234:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761556.158123:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761556.158132:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761556.158136:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761556.158145:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1482 00000100:00080000:2.0:1789761556.158153:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761556.158158:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761556.158195:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504067200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761556.158224:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761556.158228:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761556.158230:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761556.158235:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1482 00000100:00080000:2.0:1789761556.158239:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761556.158245:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761556.158255:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761556.158266:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504067200:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761556.158268:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504067328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761556.158286:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761556.158289:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761556.158290:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761556.158293:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1482 00000100:00080000:2.0:1789761556.158296:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761556.158299:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761556.158303:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761556.158311:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504067456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761556.158318:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761556.158323:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1789761556.158324:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761556.158326:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761556.158331:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1482 00000100:00080000:2.0:1789761556.158334:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761556.158337:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761556.158342:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761556.158352:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504067584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761556.158360:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761556.158362:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761556/real 1789761556] req@ffff9018d0f24700 x1876699504067200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761556.158363:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761556.158366:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761556.158374:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1482 00000100:00080000:2.0:1789761556.158379:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761556.158384:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761556.158388:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761556.158390:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504067200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761556.158399:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504067712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761556.158401:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761556.158408:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789761556.158412:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761556.158416:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504067200:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761556.158417:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761556.158422:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761556.158426:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1482 00000100:00080000:2.0:1789761556.158429:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761556.158432:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761556.158434:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761556.158442:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504067840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761556.158445:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504067328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761556.158493:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761556.158509:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761556/real 1789761556] req@ffff9018d0f26680 x1876699504067328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761556.158524:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504067328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761556.158534:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761556.158537:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761556.158540:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504067328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761556.158545:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504067456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761556.158561:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761556.158569:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761556/real 1789761556] req@ffff9018d0f24e00 x1876699504067456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761556.158577:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504067456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761556.158583:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761556.158585:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761556.158587:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504067456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761556.158590:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504067584:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761556.158601:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761556.158606:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761556/real 1789761556] req@ffff9018d0f27480 x1876699504067584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761556.158614:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504067584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761556.158618:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761556.158620:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761556.158623:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504067584:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761556.158633:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504067712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761556.158648:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761556.158658:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761556/real 1789761556] req@ffff9018d0f24000 x1876699504067712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761556.158670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504067712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761556.158679:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761556.158682:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761556.158685:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504067712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761556.158705:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504067840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761556.158726:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761556.158738:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761556/real 1789761556] req@ffff9018d0f25500 x1876699504067840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761556.158748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504067840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761556.158753:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761556.158756:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761556.158758:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504067840:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761556.798203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761557.822279:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761558.846256:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761559.870248:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761560.894214:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761561.278163:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761561.278173:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761561.278177:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761561.278196:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1487 00000100:00080000:0.0:1789761561.278209:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761561.278216:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761561.278256:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504067968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761561.278277:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761561.278280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761561.278282:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761561.278302:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1487 00000100:00080000:0.0:1789761561.278305:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761561.278308:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761561.278317:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761561.278327:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504068096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761561.278336:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761561.278338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761561.278340:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761561.278346:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1487 00000100:00100000:3.0:1789761561.278348:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504067968:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761561.278349:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761561.278355:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761561.278360:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761561.278384:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504068224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761561.278390:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761561.278393:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761561.278394:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761561.278397:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1487 00000100:00080000:0.0:1789761561.278399:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761561.278402:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761561.278407:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761561.278420:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504068352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761561.278446:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761561.278450:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761561.278452:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761561.278456:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1487 00000100:00080000:0.0:1789761561.278461:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761561.278467:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761561.278473:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761561.278481:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504068096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761561.278487:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504068480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761561.278498:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761561.278501:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761561.278503:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761561.278506:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1487 00000100:00080000:0.0:1789761561.278510:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789761561.278514:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504068224:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761561.278515:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761561.278525:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761561.278545:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504068608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761561.278548:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504068352:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761561.278581:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504068480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761561.278616:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504068608:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761561.918252:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761562.942211:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761563.966267:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761564.350342:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:1.0:1789761564.350399:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789761564.350427:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789761564.350472:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789761564.350484:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789761564.350498:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789761564.350510:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789761564.350522:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789761564.350530:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761561/real 1789761564] req@ffff9018c8b99f80 x1876699504068096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761564.350574:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504068096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761564.350591:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761564.350596:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761564.350602:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504068096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761564.350612:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761561/real 1789761564] req@ffff9018c8b9bb80 x1876699504068224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761564.350628:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504068224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761564.350637:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761564.350641:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761564.350645:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504068224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761564.350654:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761561/real 1789761564] req@ffff9018c8b98700 x1876699504068352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761564.350668:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504068352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761564.350678:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761564.350682:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761564.350686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504068352:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761564.350695:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761561/real 1789761564] req@ffff9018c8b99880 x1876699504067968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761564.350709:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504067968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761564.350717:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761564.350721:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761564.350724:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504067968:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761564.350759:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761561/real 1789761564] req@ffff9018c8b98a80 x1876699504068480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761564.350775:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504068480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761564.350784:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761564.350788:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761564.350791:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504068480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761564.350799:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761561/real 1789761564] req@ffff9018c8b98e00 x1876699504068608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761564.350827:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504068608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761564.350836:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761564.350840:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761564.350843:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504068608:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761564.990178:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761566.014262:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761566.398190:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761566.398202:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761566.398207:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761566.398218:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1492 00000100:00080000:0.0:1789761566.398229:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761566.398235:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761566.398272:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504068736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761566.398299:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761566.398303:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761566.398305:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761566.398310:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1492 00000100:00080000:0.0:1789761566.398314:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761566.398319:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761566.398327:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761566.398349:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504068736:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761566.398352:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504068864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761566.398368:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761566.398372:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761566.398374:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761566.398379:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1492 00000100:00080000:0.0:1789761566.398383:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761566.398387:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761566.398393:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761566.398405:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504068992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761566.398421:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761566.398424:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761566.398426:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761566.398430:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1492 00000100:00080000:0.0:1789761566.398434:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761566.398436:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761566.398447:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761566.398453:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761566.398468:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504069120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761566.398478:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761566.398482:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761566.398484:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761566.398488:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1492 00000100:00080000:0.0:1789761566.398493:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789761566.398497:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761566/real 1789761566] req@ffff9018c8b9b480 x1876699504068736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761566.398508:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761566.398516:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761566.398522:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504068736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761566.398533:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504069248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761566.398537:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761566.398548:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761566.398554:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761566.398557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504068736:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761566.398559:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761566.398564:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761566.398571:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1492 00000100:00080000:0.0:1789761566.398575:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761566.398580:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761566.398585:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761566.398589:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504068864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761566.398607:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504069376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761566.398621:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761566.398641:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761566/real 1789761566] req@ffff9018c8b9b800 x1876699504068864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761566.398660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504068864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761566.398668:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761566.398670:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761566.398674:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504068864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761566.398679:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504068992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761566.398699:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761566.398711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761566/real 1789761566] req@ffff9018c8b9aa00 x1876699504068992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761566.398730:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504068992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761566.398738:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761566.398742:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761566.398745:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504068992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761566.398752:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504069120:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761566.398773:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761566.398785:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761566/real 1789761566] req@ffff9018c8b98000 x1876699504069120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761566.398812:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504069120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761566.398828:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761566.398832:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761566.398835:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504069120:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761566.398842:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504069248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761566.399728:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761566.399780:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761566/real 1789761566] req@ffff9018c8b9a680 x1876699504069248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761566.399817:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504069248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761566.399827:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761566.399831:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761566.399839:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504069248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761566.399874:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504069376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761566.399897:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761566.399907:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761566/real 1789761566] req@ffff9018c8b9b100 x1876699504069376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761566.399916:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504069376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761566.399922:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761566.399925:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761566.399928:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504069376:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761567.038178:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761568.062253:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761569.086197:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761570.110213:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761571.134193:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761571.518112:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761571.518120:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761571.518124:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761571.518132:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1498 00000100:00080000:0.0:1789761571.518141:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761571.518146:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761571.518175:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504069504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761571.518200:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761571.518204:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761571.518206:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761571.518210:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1498 00000100:00080000:0.0:1789761571.518215:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761571.518220:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761571.518231:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761571.518242:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504069504:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761571.518255:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504069632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761571.518265:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761571.518268:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761571.518269:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761571.518272:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1498 00000100:00080000:0.0:1789761571.518276:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761571.518278:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761571.518282:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761571.518290:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504069760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761571.518297:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761571.518300:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761571.518301:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761571.518303:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1498 00000100:00080000:0.0:1789761571.518306:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761571.518309:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761571.518312:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761571.518317:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761571.518325:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504069888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761571.518332:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761571.518335:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761571.518336:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761571.518339:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1498 00000100:00080000:0.0:1789761571.518342:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761571.518345:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761571.518350:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761571/real 1789761571] req@ffff9018c8b98380 x1876699504069504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789761571.518351:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761571.518365:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504070016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761571.518373:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761571.518373:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504069504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761571.518381:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761571.518385:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761571.518389:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1498 00000100:00080000:0.0:1789761571.518391:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761571.518393:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761571.518400:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789761571.518402:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761571.518408:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761571.518411:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504070144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761571.518415:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504069504:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761571.518443:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504069632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761571.518462:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761571.518471:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761571/real 1789761571] req@ffff9018c8b9a300 x1876699504069632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761571.518485:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504069632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761571.518491:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761571.518493:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761571.518495:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504069632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761571.518499:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504069760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761571.518519:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761571.518531:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761571/real 1789761571] req@ffff9018c8b9ad80 x1876699504069760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761571.518545:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504069760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761571.518553:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761571.518557:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761571.518560:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504069760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761571.518567:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504069888:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761571.518583:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761571.518591:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761571/real 1789761571] req@ffff9018c8b99500 x1876699504069888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761571.518599:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504069888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761571.518604:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761571.518607:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761571.518609:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504069888:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761571.518612:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504070016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761571.518623:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761571.518628:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761571/real 1789761571] req@ffff9018c8b99c00 x1876699504070016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761571.518640:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504070016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761571.518646:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761571.518647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761571.518649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504070016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761571.518652:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504070144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761571.518661:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761571.518667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761571/real 1789761571] req@ffff9018c8b9b100 x1876699504070144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761571.518675:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504070144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761571.518680:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761571.518681:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761571.518683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504070144:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761572.158260:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761573.182194:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761574.206192:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761575.230260:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761576.254244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761576.638128:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761576.638138:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761576.638142:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761576.638152:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1503 00000100:00080000:0.0:1789761576.638163:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761576.638170:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761576.638203:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504070272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761576.638229:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761576.638232:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761576.638234:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761576.638238:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1503 00000100:00080000:0.0:1789761576.638240:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761576.638244:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761576.638253:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761576.638264:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504070400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761576.638269:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504070272:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761576.638275:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761576.638279:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761576.638280:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761576.638284:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1503 00000100:00080000:0.0:1789761576.638286:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761576.638288:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761576.638291:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761576.638301:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504070528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761576.638308:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761576.638312:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761576.638313:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761576.638317:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1503 00000100:00080000:0.0:1789761576.638321:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761576.638325:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789761576.638330:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789761576.638332:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761576.638348:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504070656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761576.638359:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761576.638363:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761576.638365:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761576.638369:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1503 00000100:00080000:0.0:1789761576.638374:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789761576.638376:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761576/real 1789761576] req@ffff9018c8b9a680 x1876699504070272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761576.638381:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761576.638389:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761576.638406:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504070272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761576.638407:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504070784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761576.638436:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761576.638440:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761576.638442:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761576.638449:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789761576.638450:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504070272:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761576.638455:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761576.638462:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1503 00000100:00080000:0.0:1789761576.638467:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761576.638472:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761576.638475:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504070400:192.168.203.158@tcp:38:kworker.0 00000008:00100000:0.0:1789761576.638480:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761576.638498:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504070912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761576.638510:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761576.638527:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761576/real 1789761576] req@ffff9018c8b98000 x1876699504070400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761576.638538:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504070400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761576.638544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761576.638547:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761576.638549:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504070400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761576.638553:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504070528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761576.638566:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761576.638573:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761576/real 1789761576] req@ffff9018c8b9aa00 x1876699504070528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761576.638581:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504070528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761576.638586:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761576.638588:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761576.638590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504070528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761576.638595:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504070656:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761576.638605:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761576.638612:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761576/real 1789761576] req@ffff9018c8b9b800 x1876699504070656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761576.638620:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504070656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761576.638625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761576.638627:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761576.638629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504070656:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761576.638632:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504070784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761576.638642:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761576.638648:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761576/real 1789761576] req@ffff9018c8b9b480 x1876699504070784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761576.638655:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504070784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761576.638661:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761576.638663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761576.638665:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504070784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761576.638675:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504070912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761576.638686:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761576.638692:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761576/real 1789761576] req@ffff9018c8b98e00 x1876699504070912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761576.638700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504070912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761576.638705:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761576.638707:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761576.638708:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504070912:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761577.278236:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761578.302222:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761579.326190:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761580.350195:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761581.374189:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761581.758186:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761581.758194:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761581.758198:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761581.758207:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1508 00000100:00080000:2.0:1789761581.758217:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761581.758227:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761581.758264:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504071040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761581.758291:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761581.758294:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761581.758296:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761581.758299:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1508 00000100:00080000:2.0:1789761581.758302:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761581.758306:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761581.758314:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761581.758325:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504071168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761581.758333:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761581.758336:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761581.758339:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761581.758341:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504071040:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761581.758343:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1508 00000100:00080000:2.0:1789761581.758347:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761581.758352:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761581.758355:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761581.758363:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504071296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761581.758370:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761581.758372:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761581.758373:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761581.758376:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1508 00000100:00080000:2.0:1789761581.758378:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761581.758380:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761581.758384:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761581.758391:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504071424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761581.758398:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761581.758400:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761581.758402:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761581.758404:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1508 00000100:00080000:2.0:1789761581.758406:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761581.758410:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789761581.758411:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789761581.758414:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761581.758422:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504071552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761581.758429:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761581.758431:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761581.758432:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761581.758435:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1508 00000100:00080000:2.0:1789761581.758437:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761581.758440:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761581.758443:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761581.758450:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504071680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761581.758457:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761581/real 1789761581] req@ffff9018d0f24a80 x1876699504071040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761581.758482:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504071040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761581.758493:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761581.758496:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761581.758499:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504071040:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761581.758529:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504071168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761581.758548:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761581.758558:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761581/real 1789761581] req@ffff9018d0f25180 x1876699504071168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761581.758567:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504071168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761581.758578:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761581.758581:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761581.758583:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504071168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761581.758587:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504071296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761581.758599:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761581.758609:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761581/real 1789761581] req@ffff9018d0f27800 x1876699504071296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761581.758622:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504071296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761581.758631:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761581.758635:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761581.758638:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504071296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761581.758645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504071424:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761581.758663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761581.758672:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761581/real 1789761581] req@ffff9018d0f24380 x1876699504071424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761581.758695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504071424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761581.758704:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761581.758708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761581.758712:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504071424:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761581.758718:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504071552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761581.758733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761581.758741:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761581/real 1789761581] req@ffff9018d0f26d80 x1876699504071552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761581.758750:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504071552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761581.758755:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761581.758757:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761581.758759:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504071552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761581.758763:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504071680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761581.758772:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761581.758778:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761581/real 1789761581] req@ffff9018d0f27100 x1876699504071680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761581.758786:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504071680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761581.758791:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761581.758792:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761581.758794:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504071680:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761582.398217:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761583.422251:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761584.446208:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761585.470259:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761586.494175:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761586.878156:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761586.878166:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761586.878178:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761586.878191:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1513 00000100:00080000:2.0:1789761586.878206:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761586.878215:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761586.878263:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504071808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761586.878294:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761586.878298:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761586.878300:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761586.878305:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1513 00000100:00080000:2.0:1789761586.878310:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761586.878316:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761586.878329:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761586.878344:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504071936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761586.878355:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504071808:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761586.878358:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761586.878368:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761586.878370:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761586.878376:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1513 00000100:00080000:2.0:1789761586.878380:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761586.878385:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761586.878391:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761586.878414:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504072064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761586.878425:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761586.878430:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761586.878432:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761586.878437:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1513 00000100:00080000:2.0:1789761586.878443:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761586.878449:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761586.878453:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761586.878460:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761586.878471:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504072192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761586.878478:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761586.878481:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761586.878483:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761586.878485:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1513 00000100:00080000:2.0:1789761586.878489:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761586.878491:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761586.878494:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761586.878501:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504072320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761586.878507:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761586.878511:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761586.878512:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761586.878516:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1513 00000100:00080000:2.0:1789761586.878518:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761586.878522:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761586.878525:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761586.878533:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504072448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761586.878730:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761586/real 1789761586] req@ffff9018d0f26a00 x1876699504071808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761586.878803:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504071808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761586.878818:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761586.878823:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761586.878828:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504071808:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761586.878859:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504071936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761586.878889:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761586.878905:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761586/real 1789761586] req@ffff9018d0f25c00 x1876699504071936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761586.878920:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504071936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761586.878930:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761586.878933:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761586.878937:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504071936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761586.878943:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504072064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761586.878967:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761586.878980:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761586/real 1789761586] req@ffff9018d0f25f80 x1876699504072064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761586.878995:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504072064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761586.879011:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761586.879014:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761586.879018:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504072064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761586.879025:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504072192:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761586.879084:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761586.879099:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761586/real 1789761586] req@ffff9018d0f26300 x1876699504072192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761586.879113:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504072192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761586.879122:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761586.879126:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761586.879130:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504072192:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761586.879136:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504072320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761586.879160:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761586.879173:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761586/real 1789761586] req@ffff9018d0f27100 x1876699504072320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761586.879186:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504072320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761586.879195:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761586.879199:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761586.879204:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504072320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761586.879211:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504072448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761586.879233:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761586.879246:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761586/real 1789761586] req@ffff9018d0f26d80 x1876699504072448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761586.879259:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504072448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761586.879268:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761586.879271:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761586.879274:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504072448:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761587.518233:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761588.542208:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761589.566235:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761590.590209:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761591.614178:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761591.998130:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761591.998138:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761591.998141:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761591.998149:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1518 00000100:00080000:0.0:1789761591.998158:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761591.998164:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761591.998194:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504072576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761591.998217:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761591.998221:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761591.998223:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761591.998227:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1518 00000100:00080000:0.0:1789761591.998232:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761591.998236:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761591.998246:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761591.998251:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504072576:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761591.998263:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504072704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761591.998272:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761591.998275:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761591.998276:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761591.998279:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1518 00000100:00080000:0.0:1789761591.998282:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761591.998284:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761591.998289:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761591.998297:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504072832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761591.998303:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761591.998306:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761591.998309:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789761591.998310:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761591.998315:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1518 00000100:00080000:0.0:1789761591.998319:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761591.998322:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761591.998326:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761591.998334:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504072960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761591.998341:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761591.998344:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761591.998346:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761591.998351:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1518 00000100:00100000:3.0:1789761591.998351:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761591/real 1789761591] req@ffff9018c8b98a80 x1876699504072576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761591.998353:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761591.998359:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761591.998365:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761591.998377:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504073088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761591.998380:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504072576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761591.998396:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761591.998399:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761591.998400:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761591.998402:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761591.998405:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761591.998409:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1518 00000100:00100000:3.0:1789761591.998413:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504072576:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761591.998414:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761591.998421:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761591.998425:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761591.998432:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504073216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761591.998438:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504072704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761591.998461:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761591.998472:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761591/real 1789761591] req@ffff9018c8b99880 x1876699504072704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761591.998480:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504072704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761591.998486:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761591.998488:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761591.998491:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504072704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761591.998495:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504072832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761591.998505:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761591.998511:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761591/real 1789761591] req@ffff9018c8b98700 x1876699504072832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761591.998518:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504072832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761591.998524:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761591.998527:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761591.998531:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504072832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761591.998537:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504072960:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761591.998556:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761591.998568:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761591/real 1789761591] req@ffff9018c8b9bb80 x1876699504072960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761591.998581:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504072960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761591.998588:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761591.998590:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761591.998593:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504072960:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761591.998597:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504073088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761591.998626:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761591.998635:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761591/real 1789761591] req@ffff9018c8b99f80 x1876699504073088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761591.998643:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504073088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761591.998649:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761591.998651:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761591.998653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504073088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761591.998665:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504073216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761591.998676:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761591.998682:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761591/real 1789761591] req@ffff9018c8b98e00 x1876699504073216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761591.998690:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504073216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761591.998694:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761591.998696:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761591.998698:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504073216:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761592.638153:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761593.662163:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761594.686177:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761595.710173:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761596.734149:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761597.118140:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761597.118148:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761597.118152:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761597.118160:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1523 00000100:00080000:2.0:1789761597.118169:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761597.118175:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761597.118205:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504073344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761597.118230:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761597.118233:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761597.118234:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761597.118237:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1523 00000100:00080000:2.0:1789761597.118239:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761597.118243:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761597.118252:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761597.118263:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504073472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761597.118263:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504073344:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761597.118277:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761597.118279:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761597.118280:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761597.118283:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1523 00000100:00080000:2.0:1789761597.118285:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761597.118287:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761597.118290:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761597.118303:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504073600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761597.118309:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761597.118312:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761597.118313:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761597.118316:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1523 00000100:00080000:2.0:1789761597.118318:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761597.118320:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761597.118323:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789761597.118330:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761597.118332:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504073728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761597.118340:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761597.118343:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761597.118344:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761597.118347:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1523 00000100:00080000:2.0:1789761597.118356:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761597.118359:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761597.118362:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761597.118370:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504073856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761597.118376:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761597/real 1789761597] req@ffff9018d0f24380 x1876699504073344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761597.118380:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761597.118387:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761597.118390:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761597.118398:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1523 00000100:00080000:2.0:1789761597.118400:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761597.118403:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761597.118406:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761597.118409:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504073344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761597.118415:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504073984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761597.118423:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761597.118429:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761597.118433:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504073344:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761597.118453:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504073472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761597.118471:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761597.118480:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761597/real 1789761597] req@ffff9018d0f27800 x1876699504073472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761597.118489:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504073472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761597.118494:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761597.118497:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761597.118500:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504073472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761597.118506:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504073600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761597.118526:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761597.118538:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761597/real 1789761597] req@ffff9018d0f25180 x1876699504073600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761597.118552:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504073600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761597.118561:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761597.118576:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761597.118580:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504073600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761597.118588:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504073728:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761597.118611:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761597.118622:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761597/real 1789761597] req@ffff9018d0f24a80 x1876699504073728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761597.118636:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504073728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761597.118644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761597.118648:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761597.118652:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504073728:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761597.118658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504073856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761597.118679:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761597.118688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761597/real 1789761597] req@ffff9018d0f25500 x1876699504073856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761597.118696:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504073856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761597.118702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761597.118705:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761597.118709:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504073856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761597.118715:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504073984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761597.118734:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761597.118747:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761597/real 1789761597] req@ffff9018d0f24000 x1876699504073984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761597.118760:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504073984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761597.118769:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761597.118773:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761597.118776:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504073984:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761597.758189:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761598.782206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761599.806187:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761600.830206:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761601.854172:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761602.238108:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761602.238121:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761602.238126:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761602.238139:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1528 00000100:00080000:2.0:1789761602.238153:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761602.238160:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761602.238208:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504074112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761602.238230:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761602.238234:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761602.238235:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761602.238239:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1528 00000100:00080000:2.0:1789761602.238241:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761602.238244:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761602.238253:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761602.238262:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504074240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761602.238269:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761602.238272:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761602.238273:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761602.238275:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1528 00000100:00080000:2.0:1789761602.238277:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761602.238280:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761602.238283:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761602.238290:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504074368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761602.238303:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761602.238305:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761602.238309:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761602.238310:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504074112:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761602.238314:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1528 00000100:00080000:2.0:1789761602.238320:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761602.238323:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761602.238327:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761602.238335:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504074496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761602.238341:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761602.238344:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761602.238346:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761602.238348:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1528 00000100:00080000:2.0:1789761602.238350:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761602.238353:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761602.238355:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761602.238363:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504074624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761602.238369:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761602.238371:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761602.238373:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761602.238375:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1528 00000100:00080000:2.0:1789761602.238385:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761602.238388:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761602.238391:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761602.238397:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761602.238401:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504074752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761602.238456:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761602/real 1789761602] req@ffff9018d0f27480 x1876699504074112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761602.238473:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504074112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761602.238482:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761602.238485:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761602.238489:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504074112:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761602.238495:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504074240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761602.238513:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761602.238522:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761602/real 1789761602] req@ffff9018d0f24e00 x1876699504074240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761602.238549:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504074240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761602.238555:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761602.238559:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761602.238562:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504074240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761602.238593:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504074368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761602.238616:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761602.238627:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761602/real 1789761602] req@ffff9018d0f26680 x1876699504074368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761602.238636:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504074368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761602.238642:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761602.238644:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761602.238646:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504074368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761602.238650:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504074496:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761602.238663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761602.238670:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761602/real 1789761602] req@ffff9018d0f24700 x1876699504074496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761602.238678:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504074496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761602.238683:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761602.238685:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761602.238686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504074496:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761602.238690:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504074624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761602.238701:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761602.238707:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761602/real 1789761602] req@ffff9018d0f24000 x1876699504074624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761602.238715:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504074624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761602.238720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761602.238722:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761602.238723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504074624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761602.238727:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504074752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761602.238736:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761602.238742:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761602/real 1789761602] req@ffff9018d0f25500 x1876699504074752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761602.238748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504074752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761602.238754:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761602.238756:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761602.238758:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504074752:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761602.878219:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761603.902140:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761604.926249:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761605.950151:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761606.974204:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761607.358110:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761607.358124:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761607.358129:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761607.358143:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1533 00000100:00080000:2.0:1789761607.358156:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761607.358165:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761607.358215:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504074880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761607.358246:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761607.358251:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761607.358254:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761607.358259:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1533 00000100:00080000:2.0:1789761607.358263:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761607.358268:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761607.358280:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761607.358293:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504074880:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761607.358297:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504075008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761607.358312:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761607.358316:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761607.358319:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761607.358323:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1533 00000100:00080000:2.0:1789761607.358327:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761607.358332:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761607.358338:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761607.358351:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504075136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761607.358361:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761607.358365:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761607.358368:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789761607.358370:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761607.358377:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1533 00000100:00080000:2.0:1789761607.358384:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761607.358389:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761607.358395:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761607.358408:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504075264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761607.358416:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761607/real 1789761607] req@ffff9018d0f24a80 x1876699504074880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761607.358425:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761607.358431:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761607.358434:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761607.358441:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1533 00000100:00100000:3.0:1789761607.358442:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504074880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761607.358447:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761607.358457:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789761607.358463:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789761607.358465:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789761607.358472:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761607.358481:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504075392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761607.358482:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504074880:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761607.358513:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761607.358517:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761607.358520:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761607.358524:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1533 00000100:00080000:2.0:1789761607.358529:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789761607.358534:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504075008:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761607.358538:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761607.358546:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761607.358562:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504075520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761607.358565:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761607.358580:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761607/real 1789761607] req@ffff9018d0f25180 x1876699504075008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761607.358591:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504075008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761607.358597:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761607.358600:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761607.358603:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504075008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761607.358609:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504075136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761607.358622:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761607.358628:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761607/real 1789761607] req@ffff9018d0f27800 x1876699504075136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761607.358636:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504075136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761607.358641:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761607.358643:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761607.358645:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504075136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761607.358648:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504075264:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761607.358660:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761607.358666:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761607/real 1789761607] req@ffff9018d0f24380 x1876699504075264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761607.358673:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504075264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761607.358678:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761607.358680:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761607.358682:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504075264:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761607.358685:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504075392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761607.358695:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761607.358701:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761607/real 1789761607] req@ffff9018d0f26d80 x1876699504075392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761607.358708:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504075392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761607.358713:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761607.358715:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761607.358717:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504075392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761607.358730:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504075520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761607.358741:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761607.358747:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761607/real 1789761607] req@ffff9018d0f27100 x1876699504075520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761607.358754:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504075520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761607.358759:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761607.358761:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761607.358763:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504075520:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761607.998185:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761609.022206:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761610.046187:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761611.070185:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761612.094165:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761612.478174:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761612.478183:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761612.478187:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761612.478196:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1539 00000100:00080000:0.0:1789761612.478207:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761612.478214:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761612.478250:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504075648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761612.478277:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761612.478281:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761612.478282:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761612.478286:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1539 00000100:00080000:0.0:1789761612.478290:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761612.478294:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761612.478305:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761612.478317:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504075648:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761612.478320:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504075776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761612.478333:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761612.478337:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761612.478339:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761612.478344:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1539 00000100:00080000:0.0:1789761612.478348:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761612.478352:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761612.478365:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761612.478371:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761612.478379:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504075904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761612.478391:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761612.478394:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761612.478397:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761612.478401:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1539 00000100:00080000:0.0:1789761612.478406:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789761612.478408:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761612/real 1789761612] req@ffff9018c8b9b480 x1876699504075648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761612.478413:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761612.478419:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761612.478436:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504075648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761612.478439:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504076032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761612.478498:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761612.478501:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761612.478506:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761612.478508:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761612.478513:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761612.478515:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504075648:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761612.478519:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1539 00000100:00080000:0.0:1789761612.478523:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761612.478531:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761612.478537:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761612.478550:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504076160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761612.478556:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504075776:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761612.478568:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761612.478573:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761612.478575:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761612.478580:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1539 00000100:00080000:0.0:1789761612.478584:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761612.478589:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789761612.478594:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789761612.478597:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761612.478609:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761612/real 1789761612] req@ffff9018c8b9b800 x1876699504075776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761612.478613:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504076288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761612.478629:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504075776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761612.478637:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761612.478639:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761612.478642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504075776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761612.478648:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504075904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761612.478664:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761612.478671:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761612/real 1789761612] req@ffff9018c8b9aa00 x1876699504075904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761612.478681:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504075904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761612.478690:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761612.478693:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761612.478697:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504075904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761612.478703:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504076032:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761612.478724:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761612.478736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761612/real 1789761612] req@ffff9018c8b98000 x1876699504076032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761612.478744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504076032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761612.478750:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761612.478752:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761612.478756:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504076032:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761612.478772:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504076160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761612.478794:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761612.478806:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761612/real 1789761612] req@ffff9018c8b9a680 x1876699504076160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761612.478816:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504076160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761612.478822:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761612.478824:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761612.478826:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504076160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761612.478830:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504076288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761612.478842:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761612.478849:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761612/real 1789761612] req@ffff9018c8b9b100 x1876699504076288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761612.478857:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504076288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761612.478863:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761612.478865:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761612.478867:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504076288:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761613.118245:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761614.142197:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761615.166206:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761616.190220:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761617.214171:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761617.598111:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761617.598123:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761617.598127:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761617.598137:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1544 00000100:00080000:0.0:1789761617.598150:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761617.598156:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761617.598194:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504076416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761617.598219:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761617.598222:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761617.598223:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761617.598226:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1544 00000100:00080000:0.0:1789761617.598228:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761617.598231:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761617.598240:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761617.598249:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504076544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761617.598257:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761617.598260:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761617.598261:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761617.598264:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1544 00000100:00080000:0.0:1789761617.598266:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761617.598268:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761617.598272:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761617.598279:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504076416:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761617.598280:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504076672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761617.598294:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761617.598297:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761617.598298:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761617.598300:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1544 00000100:00080000:0.0:1789761617.598302:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761617.598304:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761617.598307:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761617.598315:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504076800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761617.598329:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761617.598332:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761617.598333:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761617.598338:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1544 00000100:00080000:0.0:1789761617.598340:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761617.598342:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761617.598346:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761617.598358:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504076928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761617.598364:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761617.598376:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761617.598380:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761617.598383:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761617.598388:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1544 00000100:00080000:0.0:1789761617.598393:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761617.598397:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761617.598405:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761617.598430:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761617/real 1789761617] req@ffff9018c8b99c00 x1876699504076416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761617.598432:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504077056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761617.598479:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504076416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761617.598492:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761617.598497:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761617.598502:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504076416:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761617.598510:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504076544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761617.598533:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761617.598546:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761617/real 1789761617] req@ffff9018c8b99500 x1876699504076544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761617.598559:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504076544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761617.598577:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761617.598581:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761617.598585:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504076544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761617.598610:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504076672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761617.598628:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761617.598638:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761617/real 1789761617] req@ffff9018c8b9ad80 x1876699504076672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761617.598651:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504076672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761617.598660:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761617.598663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761617.598667:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504076672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761617.598673:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504076800:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761617.598690:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761617.598699:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761617/real 1789761617] req@ffff9018c8b9a300 x1876699504076800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761617.598711:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504076800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761617.598720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761617.598723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761617.598727:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504076800:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761617.598733:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504076928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761617.598751:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761617.598762:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761617/real 1789761617] req@ffff9018c8b98380 x1876699504076928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761617.598775:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504076928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761617.598784:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761617.598787:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761617.598791:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504076928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761617.598797:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504077056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761617.598816:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761617.598826:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761617/real 1789761617] req@ffff9018c8b9b100 x1876699504077056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761617.598838:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504077056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761617.598847:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761617.598850:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761617.598853:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504077056:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761618.238237:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761619.262159:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761620.286262:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761621.310202:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761622.334182:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761622.718130:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761622.718141:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761622.718146:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761622.718156:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1549 00000100:00080000:0.0:1789761622.718165:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761622.718172:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761622.718202:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504077184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761622.718225:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761622.718228:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761622.718230:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761622.718234:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1549 00000100:00080000:0.0:1789761622.718236:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761622.718240:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761622.718248:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761622.718259:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504077184:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761622.718261:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504077312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761622.718281:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761622.718284:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761622.718286:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761622.718289:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1549 00000100:00080000:0.0:1789761622.718293:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761622.718296:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761622.718301:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761622.718315:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504077440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761622.718327:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789761622.718333:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761622.718334:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761622.718339:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761622.718345:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1549 00000100:00080000:0.0:1789761622.718351:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761622.718356:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761622.718363:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761622.718388:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761622/real 1789761622] req@ffff9018c8b9a680 x1876699504077184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761622.718398:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504077568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761622.718407:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504077184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761622.718408:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761622.718416:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761622.718420:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761622.718426:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1549 00000100:00080000:2.0:1789761622.718429:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761622.718430:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761622.718435:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761622.718438:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761622.718443:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761622.718443:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504077184:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761622.718458:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504077696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761622.718465:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761622.718468:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761622.718470:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761622.718471:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504077312:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761622.718476:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1549 00000100:00080000:0.0:1789761622.718480:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761622.718483:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761622.718486:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761622.718496:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504077824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761622.718501:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761622.718520:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761622/real 1789761622] req@ffff9018c8b98000 x1876699504077312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761622.718532:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504077312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761622.718538:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761622.718541:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761622.718543:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504077312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761622.718547:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504077440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761622.718559:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761622.718565:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761622/real 1789761622] req@ffff9018c8b9aa00 x1876699504077440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761622.718577:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504077440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761622.718583:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761622.718584:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761622.718586:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504077440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761622.718590:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504077568:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761622.718602:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761622.718614:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761622/real 1789761622] req@ffff9018c8b9b800 x1876699504077568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761622.718628:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504077568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761622.718637:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761622.718641:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761622.718645:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504077568:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761622.718653:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504077696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761622.718675:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761622.718687:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761622/real 1789761622] req@ffff9018c8b9b480 x1876699504077696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761622.718696:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504077696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761622.718709:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761622.718712:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761622.718716:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504077696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761622.718732:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504077824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761622.718751:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761622.718766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761622/real 1789761622] req@ffff9018c8b98e00 x1876699504077824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761622.718782:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504077824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761622.718791:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761622.718794:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761622.718796:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504077824:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761623.358167:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761624.382165:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761625.406168:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761626.430236:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761627.454225:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761627.838125:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761627.838142:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761627.838148:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761627.838162:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1554 00000100:00080000:0.0:1789761627.838176:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761627.838186:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761627.838237:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504077952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761627.838271:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761627.838276:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761627.838278:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761627.838284:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1554 00000100:00080000:0.0:1789761627.838293:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761627.838300:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761627.838313:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761627.838327:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504077952:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761627.838346:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504078080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761627.838364:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761627.838369:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761627.838371:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761627.838377:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1554 00000100:00080000:0.0:1789761627.838384:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761627.838390:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761627.838397:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761627.838415:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504078208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761627.838428:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761627.838432:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761627.838434:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761627.838441:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1554 00000100:00080000:0.0:1789761627.838447:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761627.838449:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504078080:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761627.838457:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761627.838467:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761627.838479:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504078208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761627.838487:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504078336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761627.838508:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761627.838513:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504078336:192.168.203.158@tcp:8:kworker.0 00000100:00080000:0.0:1789761627.838516:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761627.838522:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761627.838529:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1554 00000100:00080000:0.0:1789761627.838536:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761627.838541:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761627.838548:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761627.838566:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504078464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761627.838591:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761627.838596:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761627.838598:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761627.838605:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1554 00000100:00080000:0.0:1789761627.838611:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761627.838616:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761627.838619:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504078464:192.168.203.158@tcp:38:kworker.0 00000008:00100000:0.0:1789761627.838628:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761627.838653:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504078592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761627.838691:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504078592:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761628.478213:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761629.502228:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761630.526205:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761630.910311:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:3.0:1789761630.910367:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761630.910385:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761630.910421:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761630.910436:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761630.910447:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761630.910455:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761630.910466:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:2.0:1789761630.910480:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761627/real 1789761630] req@ffff9018c8b98a80 x1876699504078464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761630.910508:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504078464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761630.910521:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761630.910527:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761630.910532:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504078464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761630.910540:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761627/real 1789761630] req@ffff9018c8b99880 x1876699504078336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761630.910552:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504078336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761630.910561:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761630.910564:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761630.910568:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504078336:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761630.910575:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761627/real 1789761630] req@ffff9018c8b9bb80 x1876699504078080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761630.910587:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504078080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761630.910601:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761630.910604:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761630.910607:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504078080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761630.910614:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761627/real 1789761630] req@ffff9018c8b98700 x1876699504078208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761630.910625:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504078208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761630.910632:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761630.910636:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761630.910640:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504078208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761630.910646:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761627/real 1789761630] req@ffff9018c8b99f80 x1876699504077952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761630.910658:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504077952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761630.910666:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761630.910669:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761630.910673:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504077952:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761630.910702:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761627/real 1789761630] req@ffff9018c8b98e00 x1876699504078592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761630.910713:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504078592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761630.910721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761630.910724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761630.910727:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504078592:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761631.550188:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761632.574233:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761632.958144:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761632.958152:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761632.958155:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761632.958166:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1559 00000100:00080000:0.0:1789761632.958183:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761632.958201:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761632.958239:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504078720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761632.958266:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761632.958269:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761632.958271:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761632.958274:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1559 00000100:00080000:0.0:1789761632.958277:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761632.958281:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761632.958289:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761632.958314:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504078848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761632.958324:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761632.958326:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504078720:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761632.958329:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761632.958334:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761632.958340:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1559 00000100:00080000:0.0:1789761632.958345:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761632.958349:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761632.958356:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761632.958369:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504078976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761632.958380:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761632.958384:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761632.958386:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761632.958390:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1559 00000800:00000100:2.0:1789761632.958396:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761632.958398:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761632.958404:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761632.958409:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761632.958419:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504079104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761632.958428:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761632.958432:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761632.958434:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761632.958438:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1559 00000100:00080000:0.0:1789761632.958442:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761632.958446:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761632.958452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761632.958468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504079232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761632.958469:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761632/real 1789761632] req@ffff9018c8b9b480 x1876699504078720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761632.958502:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761632.958506:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761632.958509:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761632.958511:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504078720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761632.958517:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1559 00000100:00080000:0.0:1789761632.958525:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761632.958531:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789761632.958532:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789761632.958540:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789761632.958541:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761632.958548:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504078720:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761632.958555:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504079360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761632.958585:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504078848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761632.958616:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761632.958629:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761632/real 1789761632] req@ffff9018c8b9b800 x1876699504078848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761632.958643:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504078848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761632.958651:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761632.958655:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761632.958659:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504078848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761632.958666:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504078976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761632.958701:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761632.958714:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761632/real 1789761632] req@ffff9018c8b9aa00 x1876699504078976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761632.958730:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504078976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761632.958739:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761632.958744:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761632.958749:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504078976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761632.958759:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504079104:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761632.958782:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761632.958796:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761632/real 1789761632] req@ffff9018c8b98000 x1876699504079104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761632.958808:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504079104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761632.958820:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761632.958824:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761632.958828:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504079104:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761632.958834:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504079232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761632.958854:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761632.958867:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761632/real 1789761632] req@ffff9018c8b9a680 x1876699504079232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761632.958878:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504079232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761632.958886:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761632.958889:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761632.958893:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504079232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761632.958900:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504079360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761632.958920:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761632.958932:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761632/real 1789761632] req@ffff9018c8b9b100 x1876699504079360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761632.958942:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504079360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761632.958957:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761632.958961:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761632.958965:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504079360:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761633.598171:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761634.622231:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761635.646245:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761636.670190:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761637.694230:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761638.078164:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761638.078178:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761638.078185:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761638.078197:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1564 00000100:00080000:2.0:1789761638.078209:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761638.078217:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761638.078292:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504079488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761638.078330:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761638.078335:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761638.078337:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761638.078343:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1564 00000100:00080000:2.0:1789761638.078348:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761638.078354:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761638.078368:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761638.078385:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504079488:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761638.078387:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504079616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761638.078408:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761638.078413:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761638.078415:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761638.078421:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1564 00000100:00080000:2.0:1789761638.078426:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761638.078431:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761638.078438:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761638.078452:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504079744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761638.078453:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761638.078477:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761638.078482:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761638.078484:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761638.078490:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761638/real 1789761638] req@ffff9018d0f26300 x1876699504079488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761638.078491:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1564 00000100:00080000:2.0:1789761638.078504:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761638.078514:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761638.078523:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761638.078524:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504079488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761638.078546:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789761638.078547:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504079872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761638.078556:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761638.078571:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504079488:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761638.078575:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761638.078582:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761638.078585:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761638.078592:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1564 00000100:00080000:2.0:1789761638.078598:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789761638.078603:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504079616:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761638.078607:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761638.078617:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761638.078638:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504080000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761638.078640:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761638.078658:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761638.078665:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761638/real 1789761638] req@ffff9018d0f25f80 x1876699504079616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761638.078666:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761638.078671:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761638.078681:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1564 00000100:00080000:2.0:1789761638.078690:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761638.078699:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761638.078704:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504079616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761638.078712:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761638.078721:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000008:00100000:2.0:1789761638.078724:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761638.078726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504079616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761638.078740:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504079744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761638.078750:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504080128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761638.078763:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761638.078778:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761638/real 1789761638] req@ffff9018d0f25c00 x1876699504079744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761638.078789:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504079744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761638.078797:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761638.078799:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761638.078802:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504079744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761638.078806:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504079872:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761638.078817:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761638.078824:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761638/real 1789761638] req@ffff9018d0f26a00 x1876699504079872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761638.078831:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504079872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761638.078836:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761638.078838:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761638.078840:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504079872:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761638.078853:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504080000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761638.078864:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761638.078871:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761638/real 1789761638] req@ffff9018d0f26300 x1876699504080000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761638.078879:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504080000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761638.078884:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761638.078886:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761638.078888:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504080000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761638.078892:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504080128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761638.078901:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761638.078907:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761638/real 1789761638] req@ffff9018d0f27100 x1876699504080128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761638.078914:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504080128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761638.078919:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761638.078921:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761638.078922:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504080128:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761638.718183:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761639.742305:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761640.766207:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761641.790189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761642.814160:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761643.198203:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761643.198218:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761643.198235:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761643.198249:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1569 00000100:00080000:0.0:1789761643.198264:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761643.198280:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761643.198333:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504080256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761643.198371:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761643.198376:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761643.198378:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761643.198384:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1569 00000100:00080000:0.0:1789761643.198389:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761643.198395:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761643.198407:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761643.198424:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504080384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761643.198424:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504080256:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761643.198439:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761643.198443:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761643.198444:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761643.198447:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1569 00000100:00080000:0.0:1789761643.198450:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761643.198452:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761643.198456:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761643.198465:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504080512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761643.198473:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761643.198475:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761643.198477:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761643.198479:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1569 00000100:00080000:0.0:1789761643.198482:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761643.198485:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761643.198488:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761643.198497:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504080640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761643.198504:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761643.198507:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761643.198510:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789761643.198511:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761643.198517:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1569 00000100:00080000:0.0:1789761643.198521:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761643.198525:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761643.198530:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761643.198537:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504080768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761643.198550:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761643.198553:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761643.198554:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761643.198557:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1569 00000100:00080000:0.0:1789761643.198561:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761643.198564:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761643.198568:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761643.198570:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761643/real 1789761643] req@ffff9018c8b98380 x1876699504080256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761643.198581:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504080896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761643.198606:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504080256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761643.198618:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761643.198622:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761643.198626:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504080256:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761643.198650:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504080384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761643.198669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761643.198680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761643/real 1789761643] req@ffff9018c8b9a300 x1876699504080384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761643.198690:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504080384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761643.198696:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761643.198699:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761643.198701:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504080384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761643.198705:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504080512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761643.198716:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761643.198722:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761643/real 1789761643] req@ffff9018c8b9ad80 x1876699504080512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761643.198730:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504080512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761643.198736:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761643.198738:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761643.198740:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504080512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761643.198744:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504080640:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789761643.198756:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761643.198762:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761643/real 1789761643] req@ffff9018c8b99500 x1876699504080640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761643.198769:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504080640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761643.198783:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761643.198785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761643.198787:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504080640:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789761643.198791:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504080768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761643.198803:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761643.198810:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761643/real 1789761643] req@ffff9018c8b99c00 x1876699504080768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761643.198818:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504080768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761643.198824:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761643.198826:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761643.198828:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504080768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761643.198831:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504080896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789761643.198841:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761643.198847:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761643/real 1789761643] req@ffff9018c8b9b100 x1876699504080896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761643.198855:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504080896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761643.198860:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761643.198861:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761643.198863:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504080896:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761643.838206:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761644.862192:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761645.886261:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761646.910211:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761647.934293:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761648.318154:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761648.318164:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761648.318169:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761648.318180:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1574 00000100:00080000:3.0:1789761648.318207:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761648.318214:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761648.318264:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504081024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761648.318301:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761648.318305:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761648.318307:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761648.318312:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1574 00000100:00080000:3.0:1789761648.318317:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761648.318333:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761648.318347:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761648.318350:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504081024:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761648.318362:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504081152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761648.318377:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761648.318380:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761648.318381:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761648.318384:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1574 00000100:00080000:3.0:1789761648.318388:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761648.318392:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761648.318397:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761648.318410:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504081280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761648.318422:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761648.318426:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761648.318428:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761648.318434:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1574 00000800:00000100:2.0:1789761648.318436:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761648.318442:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761648.318449:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761648.318456:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761648.318472:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504081408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761648.318484:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761648.318489:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789761648.318492:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761648/real 1789761648] req@ffff9018d070fb80 x1876699504081024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761648.318494:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761648.318507:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1574 00000100:00080000:3.0:1789761648.318513:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761648.318518:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504081024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761648.318521:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761648.318533:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789761648.318535:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761648.318541:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761648.318545:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504081024:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761648.318551:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504081536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761648.318564:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761648.318567:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761648.318568:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761648.318575:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504081152:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761648.318583:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1574 00000100:00080000:3.0:1789761648.318587:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761648.318591:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761648.318597:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761648.318600:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761648.318612:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504081664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761648.318615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761648/real 1789761648] req@ffff9018d070c000 x1876699504081152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761648.318638:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504081152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761648.318644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761648.318647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761648.318651:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504081152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761648.318656:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504081280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761648.318670:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761648.318680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761648/real 1789761648] req@ffff9018d070dc00 x1876699504081280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761648.318689:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504081280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761648.318694:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761648.318697:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761648.318699:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504081280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761648.318703:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504081408:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761648.318714:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761648.318720:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761648/real 1789761648] req@ffff9018d070d180 x1876699504081408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761648.318727:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504081408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761648.318737:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761648.318740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761648.318742:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504081408:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761648.318745:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504081536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761648.318763:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761648.318770:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761648/real 1789761648] req@ffff9018d070e300 x1876699504081536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761648.318777:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504081536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761648.318783:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761648.318785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761648.318787:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504081536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761648.318801:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504081664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761648.318812:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761648.318819:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761648/real 1789761648] req@ffff9018d070e680 x1876699504081664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761648.318826:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504081664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761648.318831:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761648.318833:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761648.318836:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504081664:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761648.958161:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761649.982229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761651.006239:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761652.030168:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761653.054155:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761653.438118:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761653.438127:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761653.438131:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761653.438138:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1579 00000100:00080000:3.0:1789761653.438159:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761653.438165:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761653.438196:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504081792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761653.438221:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761653.438225:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761653.438226:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761653.438229:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1579 00000100:00080000:3.0:1789761653.438232:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761653.438234:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761653.438242:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761653.438250:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504081920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761653.438258:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761653.438261:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761653.438262:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761653.438264:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1579 00000100:00080000:3.0:1789761653.438266:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761653.438271:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504081792:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761653.438272:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761653.438280:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761653.438289:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504082048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761653.438295:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761653.438298:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761653.438299:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761653.438302:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1579 00000100:00080000:3.0:1789761653.438304:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761653.438306:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761653.438311:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761653.438319:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504082176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761653.438326:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789761653.438331:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761653.438333:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761653.438337:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761653.438342:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1579 00000100:00080000:3.0:1789761653.438347:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761653.438353:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761653.438359:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761653.438372:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504082304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761653.438380:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761653/real 1789761653] req@ffff9018d070d500 x1876699504081792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761653.438384:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761653.438390:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761653.438393:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761653.438401:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1579 00000100:00080000:3.0:1789761653.438405:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761653.438408:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761653.438412:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761653.438413:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504081792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761653.438426:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761653.438427:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504082432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761653.438433:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761653.438441:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504081792:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761653.438449:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504081920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761653.438470:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761653.438480:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761653/real 1789761653] req@ffff9018d070f800 x1876699504081920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761653.438488:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504081920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761653.438494:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761653.438497:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761653.438499:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504081920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761653.438533:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504082048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761653.438556:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761653.438569:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761653/real 1789761653] req@ffff9018d070f100 x1876699504082048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761653.438583:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504082048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761653.438592:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761653.438596:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761653.438599:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504082048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761653.438606:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504082176:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761653.438635:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761653.438649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761653/real 1789761653] req@ffff9018d070e680 x1876699504082176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761653.438662:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504082176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761653.438672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761653.438676:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761653.438679:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504082176:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761653.438685:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504082304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761653.438703:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761653.438711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761653/real 1789761653] req@ffff9018d070e300 x1876699504082304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761653.438719:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504082304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761653.438725:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761653.438727:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761653.438729:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504082304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761653.438732:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504082432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761653.438743:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761653.438749:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761653/real 1789761653] req@ffff9018d070d180 x1876699504082432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761653.438756:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504082432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761653.438761:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761653.438763:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761653.438764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504082432:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761654.078188:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761655.102179:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761656.126184:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761657.150227:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761658.174188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761658.558189:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761658.558199:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761658.558205:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761658.558221:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1585 00000100:00080000:3.0:1789761658.558235:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761658.558245:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761658.558308:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504082560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761658.558337:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761658.558341:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761658.558343:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761658.558349:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1585 00000100:00080000:3.0:1789761658.558355:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761658.558360:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761658.558373:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761658.558374:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504082560:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761658.558392:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504082688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761658.558401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761658.558404:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761658.558405:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761658.558408:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1585 00000100:00080000:3.0:1789761658.558410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761658.558414:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761658.558418:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761658.558466:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761658.558501:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504082816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761658.558514:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761658.558517:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761658.558521:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761658.558522:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761658/real 1789761658] req@ffff9018d070dc00 x1876699504082560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761658.558526:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1585 00000100:00100000:0.0:1789761658.558552:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504082560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761658.558566:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761658.558572:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761658.558578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504082560:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761658.558611:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504082688:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761658.558663:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761658.558668:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761658.558673:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789761658.558675:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761658.558690:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504082944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761658.558713:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761658/real 1789761658] req@ffff9018d070c000 x1876699504082688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761658.558733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504082688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761658.558759:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761658.558763:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761658.558764:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761658.558769:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761658.558775:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789761658.558778:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1585 00000100:00080000:3.0:1789761658.558787:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761658.558789:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504082688:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761658.558795:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761658.558802:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761658.558804:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504082816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761658.558815:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504083072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761658.558824:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761658.558827:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761658.558829:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761658.558832:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1585 00000100:00080000:3.0:1789761658.558835:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761658.558839:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761658.558842:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761658.558872:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761658.558876:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761658/real 1789761658] req@ffff9018d070fb80 x1876699504082816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761658.558886:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504083200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761658.558903:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504082816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761658.558915:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761658.558920:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761658.558926:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504082816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761658.558974:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504082944:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761658.559007:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761658.559029:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761658/real 1789761658] req@ffff9018d070d880 x1876699504082944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761658.559047:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504082944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761658.559054:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761658.559057:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761658.559061:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504082944:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761658.559118:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504083072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761658.559140:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761658.559149:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761658/real 1789761658] req@ffff9018d070ce00 x1876699504083072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761658.559157:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504083072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761658.559163:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761658.559166:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761658.559168:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504083072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761658.559172:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504083200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761658.559182:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761658.559188:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761658/real 1789761658] req@ffff9018d070ca80 x1876699504083200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761658.559201:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504083200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761658.559206:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761658.559208:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761658.559210:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504083200:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761659.198206:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761660.222164:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761661.246277:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761662.270149:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761663.294219:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761663.678160:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761663.678171:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761663.678175:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761663.678186:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1590 00000100:00080000:0.0:1789761663.678198:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761663.678204:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761663.678246:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504083328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761663.678274:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761663.678276:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761663.678278:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761663.678281:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1590 00000100:00080000:0.0:1789761663.678290:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761663.678294:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761663.678303:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761663.678317:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504083456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761663.678319:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504083328:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761663.678358:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761663.678362:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761663.678365:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761663.678372:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1590 00000100:00080000:0.0:1789761663.678376:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761663.678381:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761663.678386:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761663.678398:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504083584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761663.678406:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761663.678409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761663.678411:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789761663.678418:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761663.678423:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1590 00000100:00080000:0.0:1789761663.678430:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761663.678444:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761663.678452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761663.678459:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761663/real 1789761663] req@ffff9018c8b9a680 x1876699504083328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761663.678471:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504083712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761663.678479:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504083328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761663.678511:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761663.678511:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761663.678521:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761663.678525:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789761663.678528:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504083328:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761663.678530:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761663.678540:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1590 00000100:00080000:0.0:1789761663.678545:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761663.678549:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761663.678555:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761663.678563:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504083456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761663.678569:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504083840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761663.678582:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761663.678586:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761663.678588:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761663.678592:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1590 00000800:00000100:2.0:1789761663.678596:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761663.678601:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761663.678606:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761663.678610:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761663.678615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761663/real 1789761663] req@ffff9018c8b98000 x1876699504083456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761663.678620:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504083968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761663.678651:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504083456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761663.678662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761663.678665:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761663.678669:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504083456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761663.678675:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504083584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761663.678691:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761663.678698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761663/real 1789761663] req@ffff9018c8b9aa00 x1876699504083584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761663.678711:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504083584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761663.678719:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761663.678723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761663.678726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504083584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761663.678732:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504083712:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761663.678754:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761663.678767:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761663/real 1789761663] req@ffff9018c8b9b800 x1876699504083712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761663.678777:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504083712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761663.678782:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761663.678785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761663.678787:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504083712:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761663.678800:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504083840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761663.678815:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761663.678822:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761663/real 1789761663] req@ffff9018c8b9b480 x1876699504083840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761663.678833:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504083840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761663.678848:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761663.678852:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761663.678855:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504083840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761663.678863:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504083968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761663.678884:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761663.678896:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761663/real 1789761663] req@ffff9018c8b98e00 x1876699504083968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761663.678909:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504083968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761663.678917:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761663.678921:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761663.678925:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504083968:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761664.318158:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761665.342183:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761666.366197:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761667.390212:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761668.414172:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761668.798130:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761668.798139:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761668.798143:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761668.798150:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1595 00000100:00080000:3.0:1789761668.798159:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761668.798164:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761668.798194:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504084096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761668.798216:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761668.798219:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761668.798221:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761668.798224:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1595 00000100:00080000:3.0:1789761668.798227:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761668.798229:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761668.798236:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761668.798244:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504084224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761668.798253:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761668.798257:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761668.798259:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761668.798264:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504084096:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761668.798267:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1595 00000100:00080000:3.0:1789761668.798274:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761668.798278:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761668.798284:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761668.798298:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504084352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761668.798311:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761668.798315:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761668.798317:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789761668.798321:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761668.798324:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1595 00000100:00080000:3.0:1789761668.798331:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761668.798336:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761668.798343:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761668.798356:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761668/real 1789761668] req@ffff9018d070df80 x1876699504084096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761668.798367:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504084480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761668.798373:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504084096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761668.798394:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761668.798402:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761668.798402:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761668.798407:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761668.798409:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789761668.798414:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1595 00000100:00100000:2.0:1789761668.798415:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504084096:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761668.798421:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761668.798425:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504084224:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761668.798429:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761668.798436:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761668.798447:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761668.798452:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504084608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761668.798459:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761668/real 1789761668] req@ffff9018d070ed80 x1876699504084224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761668.798472:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504084224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761668.798480:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761668.798486:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789761668.798487:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761668.798489:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504084224:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761668.798496:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761668.798499:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761668.798505:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1595 00000100:00080000:3.0:1789761668.798510:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761668.798515:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504084352:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761668.798519:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761668.798527:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761668.798533:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761668.798543:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504084736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761668.798552:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761668/real 1789761668] req@ffff9018d070f480 x1876699504084352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761668.798567:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504084352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761668.798573:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761668.798576:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761668.798578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504084352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761668.798585:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504084480:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761668.798598:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761668.798604:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761668/real 1789761668] req@ffff9018d070ca80 x1876699504084480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761668.798611:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504084480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761668.798617:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761668.798618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761668.798620:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504084480:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761668.798623:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504084608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761668.798633:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761668.798639:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761668/real 1789761668] req@ffff9018d070ce00 x1876699504084608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761668.798646:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504084608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761668.798651:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761668.798652:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761668.798654:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504084608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761668.798663:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504084736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761668.798681:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761668.798688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761668/real 1789761668] req@ffff9018d070d880 x1876699504084736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761668.798695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504084736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761668.798700:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761668.798702:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761668.798704:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504084736:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761669.438161:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761670.462181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761671.486234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761672.510206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761673.534201:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761673.918108:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761673.918117:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761673.918120:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761673.918128:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1600 00000100:00080000:2.0:1789761673.918136:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761673.918142:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761673.918172:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504084864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761673.918192:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761673.918195:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761673.918196:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761673.918199:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1600 00000100:00080000:2.0:1789761673.918202:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761673.918204:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761673.918212:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761673.918221:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504084992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761673.918229:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761673.918232:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761673.918235:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761673.918236:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504084864:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761673.918238:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1600 00000100:00080000:2.0:1789761673.918243:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761673.918247:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761673.918250:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761673.918262:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504085120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761673.918270:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761673.918272:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761673.918273:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761673.918276:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1600 00000100:00080000:2.0:1789761673.918279:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761673.918281:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761673.918284:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761673.918292:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504085248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761673.918299:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761673.918302:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761673.918303:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761673.918308:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1600 00000800:00000100:0.0:1789761673.918310:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761673.918311:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761673.918316:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761673.918319:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761673.918327:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504085376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761673.918333:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761673.918335:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761673.918336:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761673.918339:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1600 00000100:00080000:2.0:1789761673.918342:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761673.918344:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761673.918347:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761673.918355:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761673/real 1789761673] req@ffff9018d0f26d80 x1876699504084864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761673.918356:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504085504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761673.918392:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504084864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761673.918412:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761673.918416:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761673.918419:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504084864:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761673.918439:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504084992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761673.918458:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761673.918467:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761673/real 1789761673] req@ffff9018d0f24380 x1876699504084992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761673.918475:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504084992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761673.918481:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761673.918483:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761673.918485:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504084992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761673.918489:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504085120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761673.918501:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761673.918507:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761673/real 1789761673] req@ffff9018d0f27800 x1876699504085120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761673.918515:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504085120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761673.918520:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761673.918522:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761673.918523:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504085120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761673.918528:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504085248:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761673.918539:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761673.918545:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761673/real 1789761673] req@ffff9018d0f25180 x1876699504085248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761673.918552:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504085248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761673.918557:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761673.918560:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761673.918563:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504085248:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761673.918568:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504085376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761673.918588:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761673.918600:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761673/real 1789761673] req@ffff9018d0f24a80 x1876699504085376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761673.918612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504085376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761673.918617:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761673.918619:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761673.918621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504085376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761673.918626:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504085504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761673.918639:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761673.918646:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761673/real 1789761673] req@ffff9018d0f25500 x1876699504085504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761673.918654:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504085504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761673.918659:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761673.918660:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761673.918662:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504085504:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761674.558208:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761675.582201:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761676.606238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761677.630198:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761678.654248:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761679.038136:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761679.038146:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761679.038150:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761679.038161:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1605 00000100:00080000:0.0:1789761679.038170:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761679.038175:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761679.038206:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504085632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761679.038230:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761679.038232:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761679.038234:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761679.038237:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1605 00000100:00080000:0.0:1789761679.038239:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761679.038242:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761679.038250:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761679.038259:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504085760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761679.038263:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504085632:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761679.038277:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761679.038279:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761679.038280:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761679.038283:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1605 00000100:00080000:0.0:1789761679.038286:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761679.038288:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761679.038291:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761679.038307:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504085888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761679.038314:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761679.038317:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761679.038318:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761679.038321:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1605 00000100:00080000:0.0:1789761679.038323:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761679.038326:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761679.038331:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789761679.038332:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761679.038347:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504086016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761679.038359:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761679.038363:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761679.038365:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761679.038369:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1605 00000100:00080000:0.0:1789761679.038374:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761679.038380:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761679.038387:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761679.038388:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761679/real 1789761679] req@ffff9018c8b99f80 x1876699504085632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761679.038405:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504086144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761679.038409:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504085632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761679.038433:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761679.038437:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761679.038438:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761679.038442:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761679.038448:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761679.038452:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1605 00000100:00100000:2.0:1789761679.038458:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504085632:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761679.038459:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761679.038466:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761679.038471:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761679.038481:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504086272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761679.038490:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504085760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761679.038514:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761679.038524:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761679/real 1789761679] req@ffff9018c8b98700 x1876699504085760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761679.038534:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504085760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761679.038540:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761679.038542:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761679.038545:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504085760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761679.038549:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504085888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761679.038560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761679.038566:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761679/real 1789761679] req@ffff9018c8b9bb80 x1876699504085888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761679.038574:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504085888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761679.038579:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761679.038581:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761679.038584:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504085888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761679.038591:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504086016:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761679.038610:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761679.038622:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761679/real 1789761679] req@ffff9018c8b99880 x1876699504086016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761679.038637:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504086016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761679.038645:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761679.038647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761679.038650:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504086016:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761679.038654:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504086144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761679.038669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761679.038676:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761679/real 1789761679] req@ffff9018c8b98a80 x1876699504086144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761679.038684:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504086144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761679.038689:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761679.038691:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761679.038693:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504086144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761679.038706:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504086272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761679.038717:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761679.038723:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761679/real 1789761679] req@ffff9018c8b98e00 x1876699504086272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761679.038731:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504086272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761679.038736:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761679.038738:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761679.038740:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504086272:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761679.678168:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761680.702179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761681.726144:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761682.750231:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761683.774225:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761684.158127:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761684.158138:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761684.158142:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761684.158153:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1610 00000100:00080000:0.0:1789761684.158164:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761684.158169:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761684.158200:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504086400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761684.158227:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761684.158230:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761684.158231:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761684.158234:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1610 00000100:00080000:0.0:1789761684.158236:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761684.158239:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761684.158246:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761684.158255:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504086528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761684.158270:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761684.158274:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761684.158276:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761684.158282:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1610 00000100:00080000:0.0:1789761684.158287:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761684.158295:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504086400:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761684.158297:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761684.158312:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761684.158333:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504086656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761684.158346:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761684.158351:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761684.158353:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761684.158357:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1610 00000100:00080000:0.0:1789761684.158362:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761684.158365:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761684.158369:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761684.158379:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504086784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761684.158386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789761684.158395:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761684.158396:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761684.158400:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761684.158405:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1610 00000100:00080000:0.0:1789761684.158408:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761684.158413:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761684.158418:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761684.158433:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504086912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761684.158444:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761684.158448:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761684/real 1789761684] req@ffff9018c8b9b480 x1876699504086400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761684.158454:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761684.158458:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761684.158472:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1610 00000100:00100000:2.0:1789761684.158478:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504086400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761684.158481:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761684.158496:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789761684.158500:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761684.158505:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761684.158512:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504086400:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761684.158519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504086528:192.168.203.158@tcp:38:kworker.0 00000008:00100000:0.0:1789761684.158521:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761684.158544:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761684.158545:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504087040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761684.158556:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761684/real 1789761684] req@ffff9018c8b9b800 x1876699504086528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761684.158642:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504086528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761684.158650:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761684.158654:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761684.158657:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504086528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761684.158685:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504086656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761684.158699:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761684.158707:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761684/real 1789761684] req@ffff9018c8b9aa00 x1876699504086656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761684.158715:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504086656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761684.158721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761684.158723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761684.158726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504086656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761684.158731:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504086784:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761684.158750:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761684.158762:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761684/real 1789761684] req@ffff9018c8b98000 x1876699504086784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761684.158776:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504086784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761684.158790:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761684.158793:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761684.158795:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504086784:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761684.158800:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504086912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761684.158817:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761684.158824:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761684/real 1789761684] req@ffff9018c8b9a680 x1876699504086912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761684.158832:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504086912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761684.158838:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761684.158840:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761684.158842:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504086912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761684.158846:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504087040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761684.158859:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761684.158864:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761684/real 1789761684] req@ffff9018c8b9b100 x1876699504087040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761684.158876:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504087040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761684.158884:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761684.158887:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761684.158892:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504087040:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761684.798204:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761685.822180:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761686.846157:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761687.870147:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761688.894193:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761689.278113:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761689.278124:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761689.278128:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761689.278140:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1615 00000100:00080000:0.0:1789761689.278151:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761689.278158:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761689.278189:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504087168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761689.278210:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761689.278213:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761689.278215:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761689.278218:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1615 00000100:00080000:0.0:1789761689.278220:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761689.278223:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761689.278231:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761689.278247:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504087296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761689.278248:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504087168:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761689.278263:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761689.278265:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761689.278266:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761689.278269:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1615 00000100:00080000:0.0:1789761689.278272:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761689.278275:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761689.278279:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761689.278287:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504087424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761689.278294:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761689.278296:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761689.278298:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761689.278301:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1615 00000100:00080000:0.0:1789761689.278303:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761689.278305:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761689.278308:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761689.278315:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504087552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761689.278322:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789761689.278327:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761689.278328:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761689.278332:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761689.278335:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1615 00000100:00080000:0.0:1789761689.278338:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761689.278341:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761689.278344:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761689.278352:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504087680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761689.278364:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761689.278366:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761689.278370:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761689.278371:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761689/real 1789761689] req@ffff9018c8b99c00 x1876699504087168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761689.278375:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1615 00000100:00080000:0.0:1789761689.278380:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761689.278387:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761689.278391:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761689.278397:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504087168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761689.278402:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504087808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761689.278416:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761689.278421:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761689.278426:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504087168:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761689.278445:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504087296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761689.278471:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761689.278488:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761689/real 1789761689] req@ffff9018c8b99500 x1876699504087296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761689.278503:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504087296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761689.278512:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761689.278516:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761689.278519:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504087296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761689.278529:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504087424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761689.278549:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761689.278558:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761689/real 1789761689] req@ffff9018c8b9ad80 x1876699504087424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761689.278566:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504087424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761689.278572:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761689.278574:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761689.278576:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504087424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761689.278581:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504087552:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761689.278592:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761689.278598:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761689/real 1789761689] req@ffff9018c8b9a300 x1876699504087552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761689.278605:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504087552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761689.278610:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761689.278612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761689.278614:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504087552:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761689.278617:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504087680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761689.278627:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761689.278633:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761689/real 1789761689] req@ffff9018c8b98380 x1876699504087680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761689.278641:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504087680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761689.278646:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761689.278647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761689.278649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504087680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761689.278653:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504087808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761689.278662:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761689.278667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761689/real 1789761689] req@ffff9018c8b9b100 x1876699504087808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761689.278674:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504087808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761689.278679:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761689.278681:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761689.278683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504087808:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761689.918158:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761690.942157:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761691.966144:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761692.990144:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761694.014189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761694.398154:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761694.398166:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761694.398171:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761694.398183:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1620 00000100:00080000:3.0:1789761694.398202:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761694.398210:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761694.398251:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504087936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761694.398278:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761694.398282:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761694.398284:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761694.398290:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1620 00000100:00080000:3.0:1789761694.398294:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761694.398298:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761694.398308:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761694.398315:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504087936:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761694.398324:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504088064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761694.398338:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761694.398342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761694.398344:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761694.398350:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1620 00000100:00080000:3.0:1789761694.398355:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761694.398360:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761694.398367:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761694.398385:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504088192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761694.398397:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761694.398401:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761694.398403:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761694.398408:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1620 00000100:00080000:3.0:1789761694.398414:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761694.398418:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761694.398424:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761694.398437:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504088320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761694.398450:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761694.398453:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761694.398455:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761694.398460:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504088064:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761694.398462:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1620 00000100:00080000:3.0:1789761694.398470:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761694.398500:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504088192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761694.398514:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504088320:192.168.203.158@tcp:8:kworker.0 00000100:00080000:3.0:1789761694.398542:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761694.398547:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761694.398559:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504088448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761694.398571:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761694.398574:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761694.398576:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761694.398580:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1620 00000100:00080000:3.0:1789761694.398585:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761694.398590:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761694.398595:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761694.398604:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504088448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761694.398611:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504088576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761694.398647:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504088576:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761695.038213:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761696.062168:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761697.086204:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789761697.470372:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789761697.470431:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789761697.470455:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761697.470515:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761697.470526:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761697.470551:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761697.470563:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761697.470573:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789761697.470581:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761694/real 1789761697] req@ffff9018d070c000 x1876699504088064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761697.470614:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504088064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761697.470628:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761697.470634:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761697.470638:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504088064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761697.470647:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761694/real 1789761697] req@ffff9018d070dc00 x1876699504088192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761697.470660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504088192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761697.470668:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761697.470672:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761697.470676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504088192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761697.470682:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761694/real 1789761697] req@ffff9018d070d180 x1876699504088320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761697.470694:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504088320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761697.470702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761697.470705:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761697.470708:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504088320:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761697.470716:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761694/real 1789761697] req@ffff9018d070fb80 x1876699504087936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761697.470727:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504087936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761697.470736:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761697.470739:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761697.470742:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504087936:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761697.470773:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761694/real 1789761697] req@ffff9018d070e680 x1876699504088576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761697.470786:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504088576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761697.470793:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761697.470797:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761697.470801:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504088576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761697.470808:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761694/real 1789761697] req@ffff9018d070e300 x1876699504088448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761697.470839:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504088448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761697.470847:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761697.470850:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761697.470854:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504088448:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761698.110231:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761699.134189:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761699.518092:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761699.518101:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761699.518105:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761699.518113:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1626 00000100:00080000:3.0:1789761699.518123:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761699.518129:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761699.518158:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504088704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761699.518182:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761699.518186:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761699.518187:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761699.518191:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1626 00000100:00080000:3.0:1789761699.518193:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761699.518195:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761699.518204:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761699.518214:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504088832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761699.518222:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761699.518224:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761699.518226:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761699.518228:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1626 00000100:00080000:3.0:1789761699.518231:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761699.518233:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761699.518238:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504088704:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789761699.518239:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761699.518250:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504088960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761699.518258:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761699.518261:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761699.518262:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761699.518264:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1626 00000100:00080000:3.0:1789761699.518266:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761699.518268:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761699.518272:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761699.518280:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504089088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761699.518287:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761699.518289:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761699.518290:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761699.518292:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1626 00000100:00080000:3.0:1789761699.518295:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761699.518298:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761699.518299:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761699.518306:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761699.518313:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504089216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761699.518321:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761699.518323:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761699.518325:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761699.518327:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1626 00000100:00080000:3.0:1789761699.518329:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761699.518332:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761699.518336:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761699.518341:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761699/real 1789761699] req@ffff9018d070f100 x1876699504088704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761699.518346:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504089344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761699.518378:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504088704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761699.518394:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761699.518398:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761699.518400:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504088704:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761699.518406:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504088832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761699.518435:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761699.518450:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761699/real 1789761699] req@ffff9018d070f800 x1876699504088832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761699.518465:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504088832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761699.518474:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761699.518478:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761699.518481:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504088832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761699.518506:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504088960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761699.518529:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761699.518543:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761699/real 1789761699] req@ffff9018d070d500 x1876699504088960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761699.518554:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504088960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761699.518569:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761699.518572:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761699.518576:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504088960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761699.518581:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504089088:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761699.518601:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761699.518612:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761699/real 1789761699] req@ffff9018d070e300 x1876699504089088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761699.518624:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504089088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761699.518639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761699.518643:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761699.518646:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504089088:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761699.518652:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504089216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761699.518673:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761699.518685:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761699/real 1789761699] req@ffff9018d070e680 x1876699504089216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761699.518705:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504089216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761699.518714:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761699.518717:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761699.518721:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504089216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761699.518728:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504089344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761699.518750:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761699.518763:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761699/real 1789761699] req@ffff9018d070fb80 x1876699504089344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761699.518774:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504089344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761699.518782:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761699.518786:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761699.518789:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504089344:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761700.158203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761701.182188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761702.206248:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761703.230186:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761704.254185:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761704.638137:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761704.638145:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761704.638148:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761704.638156:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1631 00000100:00080000:3.0:1789761704.638166:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761704.638173:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761704.638205:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504089472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761704.638229:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761704.638231:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761704.638233:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761704.638236:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1631 00000100:00080000:3.0:1789761704.638238:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761704.638241:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761704.638254:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761704.638263:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504089600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761704.638272:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761704.638276:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789761704.638277:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504089472:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761704.638278:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761704.638286:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1631 00000100:00080000:3.0:1789761704.638289:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761704.638292:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761704.638295:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761704.638305:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504089728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761704.638312:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761704.638314:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761704.638315:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761704.638318:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1631 00000100:00080000:3.0:1789761704.638322:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761704.638325:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761704.638330:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761704.638342:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504089856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761704.638347:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761704.638363:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761704.638368:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761704.638370:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761704.638375:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1631 00000100:00080000:3.0:1789761704.638379:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761704.638382:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761704.638388:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761704.638389:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761704/real 1789761704] req@ffff9018d070d180 x1876699504089472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761704.638402:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504089984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761704.638411:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504089472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761704.638425:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761704.638428:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761704.638432:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761704.638437:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761704.638440:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1631 00000100:00080000:0.0:1789761704.638444:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789761704.638448:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761704.638449:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504089472:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761704.638458:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761704.638465:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761704.638474:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504089600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761704.638481:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504090112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761704.638503:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761704.638521:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761704/real 1789761704] req@ffff9018d070dc00 x1876699504089600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761704.638537:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504089600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761704.638546:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761704.638550:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761704.638554:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504089600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761704.638561:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504089728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761704.638583:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761704.638593:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761704/real 1789761704] req@ffff9018d070c000 x1876699504089728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761704.638601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504089728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761704.638607:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761704.638609:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761704.638611:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504089728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761704.638615:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504089856:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761704.638628:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761704.638634:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761704/real 1789761704] req@ffff9018d070d880 x1876699504089856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761704.638654:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504089856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761704.638662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761704.638666:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761704.638669:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504089856:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761704.638675:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504089984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761704.638699:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761704.638712:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761704/real 1789761704] req@ffff9018d070ce00 x1876699504089984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761704.638726:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504089984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761704.638735:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761704.638738:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761704.638742:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504089984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761704.638770:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504090112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761704.638794:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761704.638816:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761704/real 1789761704] req@ffff9018d070ca80 x1876699504090112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761704.638830:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504090112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761704.638845:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761704.638849:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761704.638852:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504090112:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761705.278217:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761706.302193:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761707.326158:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761708.350177:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761709.374184:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761709.758133:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761709.758142:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761709.758145:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761709.758153:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1636 00000100:00080000:3.0:1789761709.758162:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761709.758169:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761709.758202:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504090240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761709.758225:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761709.758228:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761709.758229:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761709.758236:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1636 00000100:00080000:3.0:1789761709.758239:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761709.758242:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761709.758251:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761709.758259:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504090368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761709.758267:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761709.758270:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761709.758271:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761709.758274:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1636 00000100:00080000:3.0:1789761709.758277:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761709.758279:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761709.758282:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761709.758285:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504090240:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761709.758291:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504090496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761709.758298:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761709.758301:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761709.758302:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761709.758305:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1636 00000100:00080000:3.0:1789761709.758306:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761709.758309:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761709.758312:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761709.758326:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504090624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761709.758333:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761709.758335:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761709.758336:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761709.758339:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1636 00000100:00080000:3.0:1789761709.758345:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761709.758348:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761709.758351:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761709.758357:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761709.758361:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504090752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761709.758368:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761709.758371:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761709.758373:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761709.758375:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1636 00000100:00080000:3.0:1789761709.758380:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761709.758382:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761709.758385:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761709.758391:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504090880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761709.758406:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761709/real 1789761709] req@ffff9018d070f480 x1876699504090240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761709.758429:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504090240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761709.758441:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761709.758446:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761709.758450:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504090240:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761709.758459:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504090368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761709.758485:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761709.758499:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761709/real 1789761709] req@ffff9018d070ed80 x1876699504090368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761709.758513:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504090368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761709.758523:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761709.758527:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761709.758531:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504090368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761709.758555:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504090496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761709.758579:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761709.758591:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761709/real 1789761709] req@ffff9018d070df80 x1876699504090496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761709.758604:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504090496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761709.758612:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761709.758616:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761709.758620:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504090496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761709.758627:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504090624:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761709.758649:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761709.758661:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761709/real 1789761709] req@ffff9018d070ca80 x1876699504090624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761709.758674:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504090624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761709.758684:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761709.758688:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761709.758692:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504090624:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761709.758698:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504090752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761709.758720:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761709.758749:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761709/real 1789761709] req@ffff9018d070ce00 x1876699504090752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761709.758763:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504090752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761709.758772:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761709.758775:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761709.758779:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504090752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761709.758785:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504090880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761709.758805:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761709.758815:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761709/real 1789761709] req@ffff9018d070d880 x1876699504090880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761709.758828:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504090880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761709.758836:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761709.758839:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761709.758843:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504090880:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761710.398171:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761711.422172:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761712.446241:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761713.470215:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761714.494202:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761714.878150:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761714.878160:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761714.878164:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761714.878174:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1641 00000100:00080000:0.0:1789761714.878186:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761714.878194:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761714.878230:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504091008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761714.878263:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761714.878268:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761714.878269:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761714.878274:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1641 00000100:00080000:0.0:1789761714.878278:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761714.878283:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761714.878292:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761714.878308:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504091136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761714.878320:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761714.878324:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761714.878325:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761714.878337:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1641 00000100:00100000:2.0:1789761714.878337:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504091008:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761714.878347:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761714.878355:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761714.878360:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761714.878376:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504091264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761714.878387:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761714.878391:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761714.878394:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761714.878400:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1641 00000100:00080000:0.0:1789761714.878405:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761714.878409:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761714.878416:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789761714.878427:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761714.878432:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504091392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761714.878449:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761714.878454:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761714.878456:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761714.878461:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1641 00000100:00080000:0.0:1789761714.878467:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761714.878472:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761714.878478:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761714.878480:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761714/real 1789761714] req@ffff9018c8b9a680 x1876699504091008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761714.878497:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504091520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761714.878504:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504091008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761714.878518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761714.878530:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761714.878533:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761714.878536:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761714.878542:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1641 00000100:00080000:2.0:1789761714.878548:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761714.878550:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761714.878557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504091008:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761714.878560:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761714.878576:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761714.878592:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504091648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761714.878599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504091136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761714.878638:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761714.878654:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761714/real 1789761714] req@ffff9018c8b98000 x1876699504091136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761714.878678:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504091136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761714.878699:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761714.878703:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761714.878707:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504091136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761714.878715:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504091264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761714.878739:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761714.878750:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761714/real 1789761714] req@ffff9018c8b9aa00 x1876699504091264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761714.878758:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504091264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761714.878764:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761714.878767:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761714.878769:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504091264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761714.878773:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504091392:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761714.878787:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761714.878794:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761714/real 1789761714] req@ffff9018c8b9b800 x1876699504091392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761714.878802:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504091392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761714.878808:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761714.878810:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761714.878811:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504091392:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761714.878815:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504091520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761714.878826:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761714.878832:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761714/real 1789761714] req@ffff9018c8b9b480 x1876699504091520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761714.878839:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504091520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761714.878844:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761714.878846:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761714.878848:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504091520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761714.878861:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504091648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761714.878872:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761714.878877:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761714/real 1789761714] req@ffff9018c8b98e00 x1876699504091648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761714.878884:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504091648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761714.878894:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761714.878896:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761714.878898:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504091648:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761715.518209:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761716.542162:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761717.566196:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761718.590139:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761719.614256:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761719.998159:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761719.998167:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761719.998172:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761719.998180:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1646 00000100:00080000:3.0:1789761719.998188:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761719.998194:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761719.998228:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504091776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761719.998254:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761719.998257:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761719.998259:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761719.998262:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1646 00000100:00080000:3.0:1789761719.998264:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761719.998267:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761719.998274:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761719.998284:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504091904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761719.998297:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761719.998300:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761719.998301:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761719.998307:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1646 00000100:00100000:2.0:1789761719.998309:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504091776:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761719.998310:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761719.998317:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761719.998322:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761719.998330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504092032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761719.998337:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761719.998340:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761719.998342:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761719.998344:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1646 00000100:00080000:3.0:1789761719.998361:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761719.998365:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761719.998369:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789761719.998373:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761719.998382:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504092160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761719.998391:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761719.998394:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761719.998395:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761719.998397:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1646 00000100:00080000:3.0:1789761719.998400:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761719.998407:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761719.998410:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761719.998421:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504092288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761719.998425:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761719/real 1789761719] req@ffff9018d070c000 x1876699504091776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761719.998440:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761719.998445:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761719.998447:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761719.998452:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1646 00000100:00100000:2.0:1789761719.998454:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504091776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761719.998455:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761719.998467:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789761719.998469:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789761719.998471:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789761719.998475:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761719.998481:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504091776:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761719.998484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504092416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761719.998505:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504091904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761719.998530:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761719.998545:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761719/real 1789761719] req@ffff9018d070dc00 x1876699504091904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761719.998559:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504091904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761719.998575:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761719.998578:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761719.998580:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504091904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761719.998593:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504092032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761719.998609:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761719.998618:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761719/real 1789761719] req@ffff9018d070d180 x1876699504092032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761719.998625:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504092032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761719.998631:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761719.998633:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761719.998635:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504092032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761719.998639:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504092160:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761719.998670:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761719.998680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761719/real 1789761719] req@ffff9018d070fb80 x1876699504092160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761719.998688:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504092160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761719.998694:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761719.998696:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761719.998699:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504092160:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761719.998703:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504092288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761719.998716:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761719.998721:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761719/real 1789761719] req@ffff9018d070e680 x1876699504092288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761719.998729:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504092288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761719.998734:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761719.998736:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761719.998738:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504092288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761719.998741:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504092416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761719.998751:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761719.998757:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761719/real 1789761719] req@ffff9018d070e300 x1876699504092416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761719.998764:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504092416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761719.998769:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761719.998771:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761719.998773:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504092416:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761720.638153:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761721.662210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761722.686138:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761723.710190:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761724.734194:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761725.118157:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761725.118167:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761725.118172:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761725.118183:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1651 00000100:00080000:3.0:1789761725.118196:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761725.118203:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761725.118251:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504092544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761725.118285:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761725.118290:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761725.118292:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761725.118297:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1651 00000100:00080000:3.0:1789761725.118300:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761725.118304:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761725.118314:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761725.118327:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504092672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761725.118335:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761725.118338:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761725.118340:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761725.118343:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1651 00000100:00080000:3.0:1789761725.118346:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761725.118353:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761725.118357:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504092544:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789761725.118359:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761725.118376:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504092800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761725.118386:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761725.118388:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761725.118389:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761725.118392:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1651 00000100:00080000:3.0:1789761725.118394:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761725.118397:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761725.118402:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761725.118412:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504092928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761725.118419:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761725.118422:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761725.118424:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761725.118430:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1651 00000800:00000100:2.0:1789761725.118434:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761725.118439:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761725.118447:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761725.118453:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761725.118470:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504093056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761725.118477:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761725/real 1789761725] req@ffff9018d070d500 x1876699504092544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761725.118513:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761725.118517:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789761725.118525:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504092544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761725.118525:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761725.118541:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1651 00000100:00080000:3.0:1789761725.118552:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761725.118556:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761725.118561:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789761725.118568:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761725.118573:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504092544:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789761725.118573:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761725.118588:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504092672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761725.118592:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504093184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761725.118629:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761725.118650:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761725/real 1789761725] req@ffff9018d070f800 x1876699504092672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761725.118660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504092672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761725.118667:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761725.118669:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761725.118671:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504092672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761725.118693:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504092800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761725.118707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761725.118714:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761725/real 1789761725] req@ffff9018d070f100 x1876699504092800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761725.118722:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504092800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761725.118727:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761725.118729:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761725.118731:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504092800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761725.118735:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504092928:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761725.118749:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761725.118760:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761725/real 1789761725] req@ffff9018d070e300 x1876699504092928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761725.118773:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504092928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761725.118781:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761725.118785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761725.118788:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504092928:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761725.118795:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504093056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761725.118811:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761725.118819:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761725/real 1789761725] req@ffff9018d070e680 x1876699504093056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761725.118826:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504093056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761725.118832:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761725.118834:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761725.118836:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504093056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761725.118840:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504093184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761725.118850:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761725.118855:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761725/real 1789761725] req@ffff9018d070fb80 x1876699504093184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761725.118862:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504093184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761725.118867:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761725.118869:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761725.118871:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504093184:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761725.758224:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761726.782224:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761727.806227:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761728.830166:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761729.854212:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761730.238146:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761730.238160:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761730.238166:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761730.238175:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1656 00000100:00080000:2.0:1789761730.238183:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761730.238189:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761730.238217:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504093312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761730.238237:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761730.238239:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761730.238241:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761730.238244:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1656 00000100:00080000:2.0:1789761730.238246:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761730.238249:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761730.238257:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761730.238269:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504093440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761730.238275:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504093312:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761730.238279:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761730.238286:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761730.238288:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761730.238292:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1656 00000100:00080000:2.0:1789761730.238295:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761730.238297:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761730.238301:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761730.238309:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504093568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761730.238315:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761730.238318:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761730.238319:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761730.238322:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1656 00000100:00080000:2.0:1789761730.238327:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761730.238330:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761730.238334:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761730.238342:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504093696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761730.238350:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761730.238352:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761730.238353:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761730.238358:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761730.238362:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1656 00000100:00080000:2.0:1789761730.238365:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761730.238368:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761730.238371:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761730.238378:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504093824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761730.238385:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761730.238387:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761730.238389:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761730.238393:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761730/real 1789761730] req@ffff9018d0f24000 x1876699504093312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761730.238394:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1656 00000100:00080000:2.0:1789761730.238407:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761730.238412:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761730.238413:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504093312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761730.238417:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761730.238427:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789761730.238432:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504093952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761730.238435:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761730.238444:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504093312:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761730.238471:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504093440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761730.238489:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761730.238498:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761730/real 1789761730] req@ffff9018d0f24700 x1876699504093440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761730.238506:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504093440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761730.238512:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761730.238515:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761730.238518:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504093440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761730.238522:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504093568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761730.238534:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761730.238540:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761730/real 1789761730] req@ffff9018d0f26680 x1876699504093568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761730.238548:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504093568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761730.238553:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761730.238555:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761730.238557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504093568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761730.238560:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504093696:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761730.238588:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761730.238598:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761730/real 1789761730] req@ffff9018d0f24e00 x1876699504093696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761730.238607:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504093696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761730.238614:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761730.238616:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761730.238619:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504093696:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761730.238623:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504093824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761730.238639:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761730.238649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761730/real 1789761730] req@ffff9018d0f27480 x1876699504093824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761730.238660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504093824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761730.238666:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761730.238669:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761730.238671:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504093824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761730.238678:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504093952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761730.238694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761730.238705:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761730/real 1789761730] req@ffff9018d0f25500 x1876699504093952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761730.238714:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504093952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761730.238720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761730.238722:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761730.238724:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504093952:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761730.878175:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761731.902203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761732.926298:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761733.950200:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761734.974158:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761735.358143:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761735.358150:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761735.358153:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761735.358161:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1661 00000100:00080000:3.0:1789761735.358170:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761735.358176:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761735.358205:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504094080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761735.358225:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761735.358228:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761735.358230:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761735.358233:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1661 00000100:00080000:3.0:1789761735.358235:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761735.358238:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761735.358245:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761735.358253:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504094208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761735.358265:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761735.358268:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761735.358269:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761735.358271:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1661 00000100:00080000:3.0:1789761735.358273:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761735.358278:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761735.358281:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504094080:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789761735.358282:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761735.358291:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504094336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761735.358298:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761735.358300:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761735.358302:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761735.358304:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1661 00000100:00080000:3.0:1789761735.358306:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761735.358308:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761735.358312:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761735.358320:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504094464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761735.358327:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761735.358330:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761735.358331:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761735.358333:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1661 00000100:00080000:3.0:1789761735.358343:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761735.358346:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761735.358350:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761735.358352:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761735.358359:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504094592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761735.358367:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761735.358369:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761735.358370:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761735.358373:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1661 00000100:00080000:3.0:1789761735.358375:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761735.358377:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761735.358380:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761735.358386:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504094720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761735.358399:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761735/real 1789761735] req@ffff9018d070d180 x1876699504094080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761735.358421:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504094080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761735.358432:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761735.358437:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761735.358449:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504094080:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761735.358458:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504094208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761735.358482:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761735.358496:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761735/real 1789761735] req@ffff9018d070dc00 x1876699504094208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761735.358510:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504094208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761735.358519:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761735.358522:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761735.358526:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504094208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761735.358562:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504094336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761735.358592:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761735.358605:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761735/real 1789761735] req@ffff9018d070c000 x1876699504094336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761735.358617:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504094336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761735.358633:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761735.358637:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761735.358641:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504094336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761735.358647:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504094464:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761735.358668:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761735.358679:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761735/real 1789761735] req@ffff9018d070d880 x1876699504094464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761735.358692:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504094464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761735.358701:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761735.358704:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761735.358708:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504094464:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761735.358713:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504094592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761735.358733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761735.358746:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761735/real 1789761735] req@ffff9018d070ce00 x1876699504094592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761735.358759:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504094592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761735.358766:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761735.358770:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761735.358774:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504094592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761735.358780:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504094720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761735.358808:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761735.358826:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761735/real 1789761735] req@ffff9018d070ca80 x1876699504094720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761735.358839:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504094720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761735.358847:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761735.358850:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761735.358853:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504094720:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761735.998142:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761737.022180:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761738.046204:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761739.070210:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761740.094218:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761740.478144:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761740.478152:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761740.478155:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761740.478163:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1667 00000100:00080000:0.0:1789761740.478172:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761740.478177:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761740.478209:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504094848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761740.478234:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761740.478237:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761740.478239:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761740.478242:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1667 00000100:00080000:0.0:1789761740.478244:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761740.478247:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761740.478255:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761740.478264:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504094976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761740.478272:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761740.478274:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761740.478276:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761740.478278:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1667 00000100:00080000:0.0:1789761740.478280:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761740.478284:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761740.478287:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761740.478293:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504095104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761740.478304:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504094848:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761740.478305:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761740.478314:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761740.478318:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761740.478323:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1667 00000100:00080000:0.0:1789761740.478326:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761740.478328:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761740.478330:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761740.478339:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504095232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761740.478350:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761740.478352:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761740.478354:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761740.478356:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1667 00000100:00080000:0.0:1789761740.478358:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761740.478362:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761740.478365:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761740.478373:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504095360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761740.478378:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761740.478381:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761740.478382:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761740.478384:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1667 00000100:00080000:0.0:1789761740.478387:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761740.478389:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761740.478393:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761740.478406:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504095488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761740.478410:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761740.478471:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761740/real 1789761740] req@ffff9018c8b98a80 x1876699504094848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761740.478489:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504094848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761740.478513:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761740.478518:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761740.478522:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504094848:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761740.478529:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504094976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761740.478551:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761740.478562:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761740/real 1789761740] req@ffff9018c8b99880 x1876699504094976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761740.478571:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504094976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761740.478577:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761740.478580:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761740.478583:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504094976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761740.478608:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504095104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761740.478621:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761740.478628:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761740/real 1789761740] req@ffff9018c8b9bb80 x1876699504095104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761740.478636:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504095104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761740.478641:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761740.478644:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761740.478646:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504095104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761740.478650:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504095232:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761740.478661:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761740.478667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761740/real 1789761740] req@ffff9018c8b98700 x1876699504095232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761740.478675:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504095232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761740.478680:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761740.478682:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761740.478684:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504095232:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761740.478689:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504095360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761740.478700:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761740.478707:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761740/real 1789761740] req@ffff9018c8b99f80 x1876699504095360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761740.478714:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504095360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761740.478719:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761740.478721:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761740.478723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504095360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761740.478726:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504095488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761740.478737:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761740.478742:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761740/real 1789761740] req@ffff9018c8b98e00 x1876699504095488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761740.478749:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504095488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761740.478753:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761740.478755:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761740.478757:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504095488:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761741.118209:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761742.142186:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761743.166268:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761744.190168:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761745.214173:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761745.598136:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761745.598147:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761745.598152:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761745.598160:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1672 00000100:00080000:2.0:1789761745.598169:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761745.598175:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761745.598205:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504095616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761745.598228:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761745.598230:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761745.598232:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761745.598235:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1672 00000100:00080000:2.0:1789761745.598237:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761745.598240:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761745.598248:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761745.598261:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504095616:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761745.598261:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504095744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761745.598277:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761745.598279:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761745.598280:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761745.598282:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1672 00000100:00080000:2.0:1789761745.598285:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761745.598287:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761745.598291:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761745.598299:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504095872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761745.598310:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761745.598312:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761745.598314:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761745.598316:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1672 00000100:00080000:2.0:1789761745.598319:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761745.598321:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761745.598325:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789761745.598329:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761745.598336:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504096000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761745.598344:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761745.598346:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761745.598348:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761745.598350:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1672 00000100:00080000:2.0:1789761745.598352:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761745.598354:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761745.598358:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761745.598369:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761745/real 1789761745] req@ffff9018d0f24a80 x1876699504095616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761745.598369:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504096128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761745.598402:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761745.598406:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761745.598409:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761745.598414:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1672 00000100:00100000:0.0:1789761745.598417:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504095616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761745.598420:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761745.598429:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789761745.598433:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789761745.598434:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761745.598441:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761745.598445:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504095616:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761745.598449:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504096256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761745.598485:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504095744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761745.598514:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761745.598527:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761745/real 1789761745] req@ffff9018d0f25180 x1876699504095744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761745.598537:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504095744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761745.598543:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761745.598546:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761745.598548:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504095744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761745.598553:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504095872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761745.598567:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761745.598573:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761745/real 1789761745] req@ffff9018d0f27800 x1876699504095872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761745.598581:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504095872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761745.598591:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761745.598594:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761745.598596:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504095872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761745.598600:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504096000:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761745.598610:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761745.598616:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761745/real 1789761745] req@ffff9018d0f24380 x1876699504096000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761745.598623:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504096000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761745.598629:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761745.598631:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761745.598632:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504096000:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761745.598636:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504096128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761745.598646:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761745.598652:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761745/real 1789761745] req@ffff9018d0f26d80 x1876699504096128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761745.598660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504096128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761745.598664:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761745.598666:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761745.598669:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504096128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761745.598672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504096256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761745.598682:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761745.598688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761745/real 1789761745] req@ffff9018d0f27100 x1876699504096256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761745.598695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504096256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761745.598700:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761745.598701:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761745.598703:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504096256:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761746.238144:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761747.262229:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761748.286248:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761749.310264:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761750.334146:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761750.718138:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761750.718146:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761750.718150:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761750.718158:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1677 00000100:00080000:3.0:1789761750.718171:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761750.718177:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761750.718208:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504096384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761750.718229:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761750.718232:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761750.718233:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761750.718236:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1677 00000100:00080000:3.0:1789761750.718238:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761750.718241:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761750.718248:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761750.718257:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504096512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761750.718265:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761750.718268:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761750.718269:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761750.718271:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1677 00000100:00080000:3.0:1789761750.718273:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761750.718276:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761750.718278:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761750.718286:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504096384:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761750.718286:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504096640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761750.718302:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761750.718304:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761750.718305:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761750.718308:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1677 00000100:00080000:3.0:1789761750.718310:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761750.718312:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761750.718316:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761750.718325:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504096768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761750.718333:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761750.718335:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761750.718336:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761750.718338:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1677 00000100:00080000:3.0:1789761750.718341:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761750.718343:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761750.718346:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761750.718353:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504096896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761750.718360:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761750.718364:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761750.718368:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761750.718369:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761750.718374:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1677 00000100:00080000:3.0:1789761750.718376:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761750.718379:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761750.718381:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761750.718387:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504097024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761750.718414:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761750/real 1789761750] req@ffff9018d070df80 x1876699504096384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761750.718449:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504096384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761750.718476:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761750.718481:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761750.718486:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504096384:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761750.718494:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504096512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761750.718522:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761750.718538:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761750/real 1789761750] req@ffff9018d070ed80 x1876699504096512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761750.718552:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504096512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761750.718561:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761750.718565:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761750.718569:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504096512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761750.718594:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504096640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761750.718617:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761750.718630:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761750/real 1789761750] req@ffff9018d070f480 x1876699504096640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761750.718643:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504096640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761750.718652:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761750.718656:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761750.718660:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504096640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761750.718667:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504096768:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761750.718688:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761750.718702:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761750/real 1789761750] req@ffff9018d070ca80 x1876699504096768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761750.718723:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504096768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761750.718733:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761750.718736:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761750.718740:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504096768:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761750.718746:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504096896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761750.718767:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761750.718775:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761750/real 1789761750] req@ffff9018d070ce00 x1876699504096896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761750.718784:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504096896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761750.718789:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761750.718791:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761750.718794:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504096896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761750.718797:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504097024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761750.718809:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761750.718815:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761750/real 1789761750] req@ffff9018d070d880 x1876699504097024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761750.718822:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504097024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761750.718832:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761750.718851:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761750.718853:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504097024:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761751.358171:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761752.382170:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761753.406208:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761754.430200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761755.454185:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761755.838137:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761755.838148:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761755.838151:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761755.838161:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1682 00000100:00080000:2.0:1789761755.838173:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761755.838181:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761755.838217:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504097152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761755.838239:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761755.838241:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761755.838243:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761755.838246:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1682 00000100:00080000:2.0:1789761755.838249:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761755.838252:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761755.838260:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761755.838269:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504097280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761755.838280:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761755.838284:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761755.838287:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761755.838292:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1682 00000100:00080000:2.0:1789761755.838296:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761755.838302:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761755.838306:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761755.838319:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504097408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761755.838327:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761755.838333:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761755.838335:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761755.838336:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504097152:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761755.838340:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1682 00000100:00080000:2.0:1789761755.838351:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761755.838356:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761755.838360:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761755.838372:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504097536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761755.838394:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761755.838396:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761755.838398:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761755.838402:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1682 00000100:00080000:2.0:1789761755.838405:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761755.838409:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761755.838414:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761755.838425:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504097664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761755.838431:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761755.838434:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761755.838435:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761755.838437:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1682 00000100:00080000:2.0:1789761755.838439:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761755.838446:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761755.838447:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761755.838459:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761755.838472:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504097792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761755.838519:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761755/real 1789761755] req@ffff9018d0f26300 x1876699504097152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761755.838544:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504097152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761755.838557:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761755.838561:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761755.838564:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504097152:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761755.838570:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504097280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761755.838592:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761755.838602:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761755/real 1789761755] req@ffff9018d0f26a00 x1876699504097280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761755.838611:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504097280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761755.838617:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761755.838619:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761755.838622:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504097280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761755.838649:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504097408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761755.838661:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761755.838668:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761755/real 1789761755] req@ffff9018d0f25c00 x1876699504097408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761755.838676:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504097408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761755.838681:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761755.838683:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761755.838685:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504097408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761755.838689:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504097536:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761755.838700:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761755.838706:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761755/real 1789761755] req@ffff9018d0f25f80 x1876699504097536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761755.838714:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504097536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761755.838719:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761755.838721:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761755.838722:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504097536:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761755.838726:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504097664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761755.838736:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761755.838743:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761755/real 1789761755] req@ffff9018d0f27100 x1876699504097664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761755.838750:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504097664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761755.838755:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761755.838757:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761755.838759:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504097664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761755.838762:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504097792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761755.838773:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761755.838778:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761755/real 1789761755] req@ffff9018d0f26d80 x1876699504097792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761755.838785:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504097792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761755.838790:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761755.838792:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761755.838794:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504097792:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761756.478246:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761757.502364:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761758.526273:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761759.550229:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761760.574225:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761760.958182:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761760.958198:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761760.958203:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761760.958213:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1687 00000100:00080000:0.0:1789761760.958225:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761760.958233:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761760.958277:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504097920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761760.958304:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761760.958308:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761760.958309:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761760.958313:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1687 00000100:00080000:0.0:1789761760.958316:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761760.958319:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761760.958329:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761760.958337:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504097920:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761760.958353:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504098048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761760.958368:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761760.958372:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761760.958375:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761760.958379:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1687 00000100:00080000:0.0:1789761760.958383:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761760.958388:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761760.958395:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761760.958408:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504098176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761760.958419:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761760.958421:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761760.958424:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761760.958427:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1687 00000100:00080000:0.0:1789761760.958431:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761760.958434:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761760.958450:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761760.958462:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504098304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761760.958469:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761760.958472:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761760.958474:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761760.958477:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504098048:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761760.958479:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1687 00000100:00080000:0.0:1789761760.958486:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761760.958489:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761760.958493:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761760.958504:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504098432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761760.958512:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761760.958512:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504098176:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761760.958521:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761760.958524:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761760.958528:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1687 00000100:00080000:0.0:1789761760.958532:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761760.958535:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761760.958539:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504098304:192.168.203.158@tcp:8:kworker.0 00000008:00100000:0.0:1789761760.958542:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761760.958556:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504098560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761760.958578:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504098432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761760.958599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504098560:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761761.598206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761762.622293:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761763.646197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789761764.030447:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789761764.030510:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789761764.030539:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761764.030592:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761764.030603:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761764.030613:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761764.030624:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761764.030635:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789761764.030652:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761760/real 1789761764] req@ffff9018c8b9a680 x1876699504098432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761764.030686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504098432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761764.030700:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761764.030706:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761764.030712:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504098432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761764.030721:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761760/real 1789761764] req@ffff9018c8b9b800 x1876699504098048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761764.030736:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504098048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761764.030746:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761764.030750:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761764.030754:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504098048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761764.030762:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761760/real 1789761764] req@ffff9018c8b9aa00 x1876699504098176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761764.030776:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504098176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761764.030785:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761764.030790:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761764.030793:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504098176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761764.030801:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761760/real 1789761764] req@ffff9018c8b98000 x1876699504098304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761764.030814:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504098304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761764.030822:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761764.030836:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761764.030840:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504098304:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761764.030848:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761760/real 1789761764] req@ffff9018c8b9b480 x1876699504097920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761764.030862:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504097920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761764.030871:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761764.030875:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761764.030879:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504097920:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761764.030915:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761760/real 1789761764] req@ffff9018c8b9b100 x1876699504098560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761764.030931:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504098560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761764.030940:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761764.030945:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761764.030948:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504098560:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761764.670190:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761765.694230:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761766.078165:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761766.078174:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761766.078178:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761766.078208:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1692 00000100:00080000:0.0:1789761766.078217:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761766.078223:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761766.078256:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504098688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761766.078287:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761766.078291:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761766.078292:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761766.078296:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1692 00000100:00080000:0.0:1789761766.078311:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761766.078317:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761766.078328:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761766.078341:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504098688:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761766.078351:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504098816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761766.078368:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761766.078371:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761766.078373:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761766.078378:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1692 00000100:00080000:0.0:1789761766.078384:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761766.078389:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761766.078395:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761766.078411:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504098944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761766.078422:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761766.078424:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761766.078425:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761766.078428:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1692 00000100:00080000:0.0:1789761766.078432:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761766.078435:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761766.078441:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761766.078455:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504099072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761766.078462:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761766.078476:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761766.078481:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761766.078486:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761766.078493:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1692 00000100:00080000:0.0:1789761766.078499:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761766.078505:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761766.078514:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761766.078530:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761766/real 1789761766] req@ffff9018c8b9b100 x1876699504098688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761766.078535:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504099200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761766.078573:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761766.078578:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504098688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761766.078579:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761766.078585:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761766.078601:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1692 00000100:00080000:0.0:1789761766.078610:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761766.078614:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761766.078616:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789761766.078631:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:0.0:1789761766.078632:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761766.078644:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504098688:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761766.078656:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504099328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761766.078693:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504098816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761766.078731:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761766.078755:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761766/real 1789761766] req@ffff9018c8b9b480 x1876699504098816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761766.078769:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504098816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761766.078777:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761766.078781:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761766.078784:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504098816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761766.078792:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504098944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761766.078812:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761766.078824:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761766/real 1789761766] req@ffff9018c8b98000 x1876699504098944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761766.078836:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504098944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761766.078845:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761766.078849:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761766.078852:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504098944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761766.078859:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504099072:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761766.078882:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761766.078893:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761766/real 1789761766] req@ffff9018c8b9aa00 x1876699504099072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761766.078912:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504099072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761766.078921:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761766.078925:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761766.078928:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504099072:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761766.078936:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504099200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761766.078958:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761766.078969:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761766/real 1789761766] req@ffff9018c8b9b800 x1876699504099200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761766.078980:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504099200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761766.078987:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761766.078990:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761766.078994:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504099200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761766.079000:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504099328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761766.079022:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761766.079032:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761766/real 1789761766] req@ffff9018c8b9a680 x1876699504099328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761766.079076:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504099328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761766.079093:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761766.079097:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761766.079101:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504099328:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761766.718342:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761767.742233:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761768.766269:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761769.790175:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761770.814257:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761771.198146:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761771.198162:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761771.198168:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761771.198182:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1697 00000100:00080000:2.0:1789761771.198207:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761771.198217:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761771.198289:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504099456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761771.198316:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761771.198320:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761771.198321:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761771.198325:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1697 00000100:00080000:2.0:1789761771.198328:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761771.198332:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761771.198343:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761771.198362:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504099584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761771.198363:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504099456:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761771.198382:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761771.198385:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761771.198387:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761771.198392:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1697 00000100:00080000:2.0:1789761771.198396:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761771.198399:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761771.198403:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761771.198414:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504099712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761771.198421:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761771.198424:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761771.198426:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761771.198429:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1697 00000100:00080000:2.0:1789761771.198432:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761771.198434:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761771.198438:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789761771.198445:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761771.198447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504099840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761771.198456:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761771.198459:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761771.198460:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761771.198463:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1697 00000100:00080000:2.0:1789761771.198466:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761771.198469:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761771.198472:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761771.198478:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504099968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761771.198484:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761771.198486:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761771.198487:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761771.198490:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1697 00000100:00080000:2.0:1789761771.198492:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761771.198495:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761771.198498:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761771.198505:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761771/real 1789761771] req@ffff9018d0f24380 x1876699504099456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761771.198507:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504100096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761771.198551:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504099456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761771.198566:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761771.198571:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761771.198576:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504099456:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761771.198606:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504099584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761771.198635:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761771.198652:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761771/real 1789761771] req@ffff9018d0f27800 x1876699504099584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761771.198665:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504099584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761771.198675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761771.198679:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761771.198683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504099584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761771.198690:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504099712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761771.198711:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761771.198720:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761771/real 1789761771] req@ffff9018d0f25180 x1876699504099712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761771.198729:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504099712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761771.198734:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761771.198737:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761771.198740:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504099712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761771.198754:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504099840:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761771.198777:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761771.198790:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761771/real 1789761771] req@ffff9018d0f24a80 x1876699504099840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761771.198802:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504099840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761771.198808:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761771.198811:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761771.198813:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504099840:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761771.198817:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504099968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761771.198832:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761771.198839:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761771/real 1789761771] req@ffff9018d0f25500 x1876699504099968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761771.198847:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504099968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761771.198853:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761771.198854:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761771.198856:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504099968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761771.198860:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504100096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761771.198871:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761771.198876:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761771/real 1789761771] req@ffff9018d0f27480 x1876699504100096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761771.198884:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504100096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761771.198889:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761771.198891:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761771.198893:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504100096:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761771.838181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761772.862238:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761773.886262:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761774.910188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761775.934269:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761776.318180:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761776.318195:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761776.318201:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761776.318214:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1702 00000100:00080000:3.0:1789761776.318241:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761776.318252:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761776.318312:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504100224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761776.318349:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761776.318354:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761776.318355:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761776.318361:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1702 00000100:00080000:3.0:1789761776.318366:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761776.318371:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761776.318384:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761776.318401:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504100352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761776.318410:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504100224:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761776.318413:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761776.318420:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761776.318423:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761776.318428:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1702 00000100:00080000:3.0:1789761776.318433:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761776.318437:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761776.318444:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761776.318459:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504100480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761776.318477:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761776.318481:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761776.318484:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761776.318485:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761776.318494:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1702 00000100:00080000:3.0:1789761776.318500:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761776.318505:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761776.318512:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761776.318527:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504100608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761776.318538:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761776.318544:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789761776.318547:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761776/real 1789761776] req@ffff9018d070c000 x1876699504100224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761776.318547:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761776.318556:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1702 00000100:00080000:3.0:1789761776.318564:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761776.318572:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761776.318578:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761776.318581:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504100224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761776.318594:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761776.318595:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504100736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761776.318601:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761776.318605:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504100224:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761776.318617:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761776.318620:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761776.318622:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761776.318626:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1702 00000100:00080000:3.0:1789761776.318630:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761776.318633:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504100352:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761776.318637:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761776.318644:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761776.318656:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504100864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761776.318663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761776.318681:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761776/real 1789761776] req@ffff9018d070dc00 x1876699504100352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761776.318691:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504100352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761776.318697:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761776.318700:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761776.318702:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504100352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761776.318706:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504100480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761776.318719:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761776.318725:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761776/real 1789761776] req@ffff9018d070d180 x1876699504100480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761776.318733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504100480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761776.318738:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761776.318740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761776.318742:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504100480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761776.318745:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504100608:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761776.318756:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761776.318762:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761776/real 1789761776] req@ffff9018d070fb80 x1876699504100608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761776.318769:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504100608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761776.318775:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761776.318777:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761776.318778:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504100608:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761776.318782:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504100736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761776.318791:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761776.318797:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761776/real 1789761776] req@ffff9018d070e680 x1876699504100736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761776.318810:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504100736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761776.318815:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761776.318817:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761776.318818:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504100736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761776.318830:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504100864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761776.318841:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761776.318847:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761776/real 1789761776] req@ffff9018d070e300 x1876699504100864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761776.318854:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504100864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761776.318859:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761776.318861:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761776.318863:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504100864:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761776.958186:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761777.982189:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761779.006135:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761780.030258:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761781.054219:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761781.438131:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761781.438141:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761781.438145:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761781.438155:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1707 00000100:00080000:3.0:1789761781.438165:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761781.438171:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761781.438217:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504100992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761781.438240:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761781.438244:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761781.438246:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761781.438250:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1707 00000100:00080000:3.0:1789761781.438255:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761781.438259:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761781.438268:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761781.438280:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504100992:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761781.438283:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504101120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761781.438293:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761781.438295:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761781.438297:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761781.438299:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1707 00000100:00080000:3.0:1789761781.438302:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761781.438304:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761781.438308:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761781.438316:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504101248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761781.438322:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761781.438325:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761781.438326:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761781.438329:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1707 00000100:00080000:3.0:1789761781.438333:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761781.438336:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761781.438340:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789761781.438343:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761781.438357:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504101376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761781.438367:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761781.438371:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761781.438373:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761781.438377:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1707 00000100:00080000:3.0:1789761781.438382:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761781.438386:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761781/real 1789761781] req@ffff9018d070f100 x1876699504100992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761781.438387:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761781.438393:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761781.438405:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504101504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761781.438413:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504100992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761781.438413:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761781.438419:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761781.438421:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761781.438426:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1707 00000100:00080000:3.0:1789761781.438430:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761781.438433:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789761781.438435:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789761781.438436:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761781.438444:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789761781.438445:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504101632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761781.438454:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504100992:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761781.438479:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504101120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761781.438497:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761781.438505:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761781/real 1789761781] req@ffff9018d070f800 x1876699504101120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761781.438513:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504101120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761781.438518:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761781.438520:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761781.438522:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504101120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761781.438525:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504101248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761781.438535:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761781.438544:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761781/real 1789761781] req@ffff9018d070d500 x1876699504101248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761781.438551:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504101248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761781.438556:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761781.438558:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761781.438560:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504101248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761781.438564:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504101376:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761781.438575:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761781.438584:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761781/real 1789761781] req@ffff9018d070e300 x1876699504101376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761781.438595:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504101376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761781.438602:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761781.438605:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761781.438608:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504101376:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761781.438614:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504101504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761781.438638:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761781.438646:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761781/real 1789761781] req@ffff9018d070e680 x1876699504101504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761781.438653:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504101504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761781.438658:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761781.438660:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761781.438663:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504101504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761781.438668:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504101632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761781.438685:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761781.438695:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761781/real 1789761781] req@ffff9018d070fb80 x1876699504101632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761781.438712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504101632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761781.438720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761781.438722:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761781.438724:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504101632:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761782.078176:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761783.102262:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761784.126199:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761785.150209:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761786.174213:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761786.558106:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761786.558116:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761786.558119:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761786.558128:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1713 00000100:00080000:3.0:1789761786.558137:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761786.558143:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761786.558189:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504101760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761786.558213:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761786.558216:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761786.558218:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761786.558221:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1713 00000100:00080000:3.0:1789761786.558226:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761786.558230:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761786.558239:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761786.558249:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504101760:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761786.558251:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504101888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761786.558264:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761786.558268:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761786.558270:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761786.558273:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1713 00000100:00080000:3.0:1789761786.558277:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761786.558280:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761786.558284:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761786.558295:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504102016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761786.558304:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761786.558310:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761786.558311:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761786.558314:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761786.558316:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1713 00000100:00080000:3.0:1789761786.558321:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761786.558325:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761786.558330:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761786.558341:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504102144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761786.558357:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761786.558360:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761786.558362:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761786.558366:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761786/real 1789761786] req@ffff9018d070d180 x1876699504101760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761786.558368:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1713 00000100:00080000:3.0:1789761786.558374:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761786.558381:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761786.558386:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761786.558389:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504101760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761786.558398:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761786.558398:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504102272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761786.558406:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761786.558412:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504101760:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761786.558416:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761786.558420:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761786.558421:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761786.558426:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1713 00000100:00080000:3.0:1789761786.558428:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761786.558432:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761786.558436:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504101888:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789761786.558438:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761786.558456:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504102400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761786.558466:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761786.558482:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761786/real 1789761786] req@ffff9018d070dc00 x1876699504101888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761786.558495:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504101888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761786.558508:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761786.558512:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761786.558515:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504101888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761786.558521:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504102016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761786.558541:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761786.558553:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761786/real 1789761786] req@ffff9018d070c000 x1876699504102016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761786.558565:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504102016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761786.558573:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761786.558576:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761786.558579:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504102016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761786.558585:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504102144:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761786.558604:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761786.558627:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761786/real 1789761786] req@ffff9018d070d880 x1876699504102144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761786.558639:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504102144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761786.558646:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761786.558649:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761786.558653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504102144:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761786.558658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504102272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761786.558678:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761786.558689:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761786/real 1789761786] req@ffff9018d070ce00 x1876699504102272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761786.558701:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504102272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761786.558708:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761786.558710:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761786.558712:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504102272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761786.558724:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504102400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761786.558737:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761786.558744:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761786/real 1789761786] req@ffff9018d070ca80 x1876699504102400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761786.558751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504102400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761786.558755:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761786.558757:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761786.558758:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504102400:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761787.198163:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761788.222154:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761789.246198:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761790.270172:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761791.294212:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761791.678225:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761791.678237:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761791.678241:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761791.678252:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1718 00000100:00080000:3.0:1789761791.678266:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761791.678274:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761791.678320:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504102528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761791.678351:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761791.678356:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761791.678358:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761791.678366:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1718 00000100:00080000:3.0:1789761791.678371:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761791.678377:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761791.678392:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761791.678396:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504102528:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761791.678407:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504102656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761791.678418:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761791.678421:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761791.678422:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761791.678427:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1718 00000100:00080000:3.0:1789761791.678430:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761791.678433:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761791.678437:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761791.678447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504102784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761791.678454:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761791.678458:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761791.678459:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761791.678462:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1718 00000100:00080000:3.0:1789761791.678473:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761791.678476:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761791.678480:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789761791.678483:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761791.678500:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504102912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761791.678511:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761791.678514:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761791.678515:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761791.678518:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1718 00000100:00080000:3.0:1789761791.678523:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761791.678528:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761791.678532:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761791.678539:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761791/real 1789761791] req@ffff9018d070f480 x1876699504102528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761791.678548:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504103040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761791.678566:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761791.678574:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789761791.678575:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504102528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761791.678579:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761791.678592:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1718 00000100:00080000:0.0:1789761791.678606:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761791.678606:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761791.678622:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789761791.678622:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761791.678632:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504102528:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789761791.678633:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761791.678656:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504103168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761791.678672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504102656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761791.678694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761791.678706:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761791/real 1789761791] req@ffff9018d070ed80 x1876699504102656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761791.678715:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504102656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761791.678721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761791.678723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761791.678725:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504102656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761791.678731:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504102784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761791.678742:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761791.678748:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761791/real 1789761791] req@ffff9018d070df80 x1876699504102784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761791.678761:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504102784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761791.678767:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761791.678769:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761791.678771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504102784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761791.678775:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504102912:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761791.678791:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761791.678803:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761791/real 1789761791] req@ffff9018d070ca80 x1876699504102912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761791.678817:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504102912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761791.678825:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761791.678829:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761791.678832:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504102912:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761791.678839:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504103040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761791.678860:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761791.678871:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761791/real 1789761791] req@ffff9018d070ce00 x1876699504103040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761791.678883:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504103040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761791.678897:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761791.678901:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761791.678904:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504103040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761791.678941:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504103168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761791.678963:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761791.678974:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761791/real 1789761791] req@ffff9018d070d880 x1876699504103168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761791.678986:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504103168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761791.678994:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761791.678997:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761791.679000:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504103168:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761792.318256:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761793.342289:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761794.366272:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761795.390196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761796.414190:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761796.798100:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761796.798111:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761796.798115:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761796.798126:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1723 00000100:00080000:3.0:1789761796.798138:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761796.798145:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761796.798197:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504103296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761796.798225:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761796.798228:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761796.798230:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761796.798233:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1723 00000100:00080000:3.0:1789761796.798237:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761796.798241:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761796.798251:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761796.798267:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504103296:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761796.798267:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504103424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761796.798283:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761796.798285:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761796.798287:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761796.798292:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1723 00000100:00080000:3.0:1789761796.798297:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761796.798302:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761796.798308:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761796.798323:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504103552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761796.798336:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761796.798339:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761796.798342:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761796.798347:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761796.798351:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1723 00000100:00080000:3.0:1789761796.798358:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761796.798363:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761796.798370:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761796.798385:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504103680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761796.798395:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761796.798398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761796.798400:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761796.798403:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1723 00000100:00100000:0.0:1789761796.798407:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761796/real 1789761796] req@ffff9018d070c000 x1876699504103296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761796.798410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761796.798421:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761796.798429:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761796.798430:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504103296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761796.798443:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761796.798449:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504103808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761796.798450:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761796.798457:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504103296:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761796.798493:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761796.798496:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504103424:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761796.798498:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761796.798503:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761796.798509:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1723 00000100:00080000:3.0:1789761796.798514:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761796.798517:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789761796.798522:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789761796.798523:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761796.798534:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761796/real 1789761796] req@ffff9018d070dc00 x1876699504103424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761796.798543:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504103936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761796.798546:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504103424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761796.798570:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761796.798573:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761796.798576:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504103424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761796.798591:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504103552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761796.798605:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761796.798612:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761796/real 1789761796] req@ffff9018d070d180 x1876699504103552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761796.798619:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504103552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761796.798624:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761796.798626:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761796.798628:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504103552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761796.798632:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504103680:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761796.798643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761796.798658:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761796/real 1789761796] req@ffff9018d070fb80 x1876699504103680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761796.798665:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504103680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761796.798670:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761796.798672:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761796.798674:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504103680:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761796.798677:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504103808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761796.798686:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761796.798692:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761796/real 1789761796] req@ffff9018d070e680 x1876699504103808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761796.798700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504103808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761796.798705:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761796.798707:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761796.798708:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504103808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761796.798720:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504103936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761796.798730:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761796.798736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761796/real 1789761796] req@ffff9018d070e300 x1876699504103936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761796.798748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504103936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761796.798753:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761796.798755:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761796.798764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504103936:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761797.438280:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761798.462221:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761799.486242:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761800.510270:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761801.534301:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761801.918138:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761801.918152:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761801.918165:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761801.918175:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1728 00000100:00080000:3.0:1789761801.918187:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761801.918195:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761801.918236:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504104064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761801.918266:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761801.918270:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761801.918271:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761801.918275:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1728 00000100:00080000:3.0:1789761801.918277:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761801.918280:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761801.918290:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761801.918301:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504104192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761801.918308:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761801.918311:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761801.918312:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761801.918316:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1728 00000100:00080000:3.0:1789761801.918318:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761801.918320:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761801.918328:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761801.918340:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504104320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761801.918344:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504104064:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761801.918349:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761801.918355:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761801.918360:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761801.918365:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1728 00000100:00080000:3.0:1789761801.918367:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761801.918370:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761801.918373:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761801.918388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504104448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761801.918396:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761801.918399:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761801.918401:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761801.918406:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1728 00000100:00080000:3.0:1789761801.918409:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761801.918414:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761801.918421:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761801.918428:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761801.918437:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504104576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761801.918451:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761801.918455:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761801.918458:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761801.918481:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1728 00000100:00080000:3.0:1789761801.918487:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761801.918495:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761801.918498:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761801/real 1789761801] req@ffff9018d070d500 x1876699504104064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789761801.918501:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761801.918521:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504104704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761801.918550:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504104064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761801.918561:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761801.918565:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761801.918568:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504104064:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761801.918576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504104192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761801.918602:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761801.918618:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761801/real 1789761801] req@ffff9018d070f800 x1876699504104192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761801.918636:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504104192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761801.918653:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761801.918657:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761801.918661:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504104192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761801.918688:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504104320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761801.918709:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761801.918718:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761801/real 1789761801] req@ffff9018d070f100 x1876699504104320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761801.918732:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504104320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761801.918738:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761801.918742:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761801.918749:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504104320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761801.918757:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504104448:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761801.918774:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761801.918784:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761801/real 1789761801] req@ffff9018d070e300 x1876699504104448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761801.918791:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504104448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761801.918798:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761801.918803:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761801.918808:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504104448:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761801.918817:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504104576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761801.918844:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761801.918857:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761801/real 1789761801] req@ffff9018d070e680 x1876699504104576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761801.918865:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504104576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761801.918870:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761801.918872:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761801.918875:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504104576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761801.918879:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504104704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761801.918894:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761801.918901:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761801/real 1789761801] req@ffff9018d070fb80 x1876699504104704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761801.918914:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504104704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761801.918919:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761801.918921:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761801.918924:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504104704:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761802.558161:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761803.582240:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761804.606218:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761805.630213:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761806.654176:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761807.038101:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761807.038112:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761807.038117:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761807.038127:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1733 00000100:00080000:3.0:1789761807.038139:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761807.038146:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761807.038188:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504104832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761807.038219:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761807.038222:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761807.038223:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761807.038226:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1733 00000100:00080000:3.0:1789761807.038229:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761807.038231:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761807.038241:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761807.038251:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504104960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761807.038259:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761807.038262:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761807.038264:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761807.038270:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504104832:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761807.038271:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1733 00000100:00080000:3.0:1789761807.038282:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761807.038288:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761807.038292:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761807.038302:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504105088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761807.038309:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761807.038313:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761807.038314:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761807.038317:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1733 00000100:00080000:3.0:1789761807.038322:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761807.038328:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761807.038331:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761807.038340:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504105216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761807.038346:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761807.038349:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761807.038352:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761807.038357:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761807.038360:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1733 00000100:00080000:3.0:1789761807.038364:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761807.038368:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761807.038371:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761807.038380:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504105344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761807.038386:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761807.038389:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761807.038391:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761807.038396:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1733 00000100:00080000:3.0:1789761807.038400:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761807.038403:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761807.038408:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761807.038412:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761807/real 1789761807] req@ffff9018d070d180 x1876699504104832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761807.038421:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504105472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761807.038445:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504104832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761807.038466:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761807.038471:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761807.038483:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504104832:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761807.038505:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504104960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761807.038525:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761807.038534:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761807/real 1789761807] req@ffff9018d070dc00 x1876699504104960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761807.038543:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504104960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761807.038548:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761807.038551:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761807.038553:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504104960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761807.038557:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504105088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761807.038569:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761807.038579:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761807/real 1789761807] req@ffff9018d070c000 x1876699504105088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761807.038599:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504105088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761807.038608:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761807.038611:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761807.038615:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504105088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761807.038622:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504105216:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761807.038640:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761807.038648:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761807/real 1789761807] req@ffff9018d070d880 x1876699504105216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761807.038656:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504105216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761807.038662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761807.038664:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761807.038666:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504105216:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761807.038670:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504105344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761807.038681:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761807.038686:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761807/real 1789761807] req@ffff9018d070ce00 x1876699504105344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761807.038694:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504105344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761807.038703:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761807.038705:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761807.038707:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504105344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761807.038710:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504105472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761807.038720:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761807.038726:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761807/real 1789761807] req@ffff9018d070ca80 x1876699504105472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761807.038733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504105472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761807.038737:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761807.038739:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761807.038741:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504105472:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761807.678159:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761808.702162:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761809.726178:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761810.750145:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761811.774157:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761812.158144:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761812.158159:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761812.158163:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761812.158174:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1738 00000100:00080000:2.0:1789761812.158183:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761812.158189:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761812.158220:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504105600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761812.158242:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761812.158245:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761812.158247:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761812.158252:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1738 00000100:00080000:2.0:1789761812.158255:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761812.158259:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761812.158269:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761812.158284:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504105728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761812.158291:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504105600:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761812.158303:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761812.158306:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761812.158307:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761812.158311:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1738 00000100:00080000:2.0:1789761812.158314:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761812.158317:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761812.158321:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761812.158334:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504105856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761812.158344:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761812.158348:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761812.158350:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761812.158352:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761812.158359:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1738 00000100:00080000:2.0:1789761812.158364:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761812.158369:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761812.158375:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761812.158388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504105984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761812.158391:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761812/real 1789761812] req@ffff9018d0f24e00 x1876699504105600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761812.158420:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761812.158424:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504105600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761812.158426:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761812.158431:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761812.158438:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761812.158440:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1738 00000100:00080000:0.0:1789761812.158445:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789761812.158446:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761812.158449:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504105600:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761812.158457:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761812.158463:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761812.158475:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504106112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761812.158480:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504105728:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761812.158493:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761812.158497:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761812.158500:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761812.158504:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789761812.158506:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1738 00000100:00080000:2.0:1789761812.158512:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761812.158517:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761812.158520:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761812/real 1789761812] req@ffff9018d0f26680 x1876699504105728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761812.158524:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761812.158534:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504105728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761812.158543:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789761812.158543:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504106240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761812.158548:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761812.158555:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504105728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761812.158569:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504105856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761812.158593:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761812.158614:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761812/real 1789761812] req@ffff9018d0f24700 x1876699504105856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761812.158626:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504105856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761812.158632:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761812.158634:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761812.158636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504105856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761812.158640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504105984:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761812.158656:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761812.158663:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761812/real 1789761812] req@ffff9018d0f24000 x1876699504105984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761812.158670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504105984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761812.158675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761812.158677:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761812.158679:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504105984:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761812.158689:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504106112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761812.158699:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761812.158705:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761812/real 1789761812] req@ffff9018d0f24e00 x1876699504106112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761812.158712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504106112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761812.158718:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761812.158719:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761812.158721:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504106112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761812.158724:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504106240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761812.158733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761812.158739:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761812/real 1789761812] req@ffff9018d0f27480 x1876699504106240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761812.158750:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504106240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761812.158757:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761812.158759:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761812.158762:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504106240:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761812.798217:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761813.822159:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761814.846251:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761815.870192:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761816.894223:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761817.278201:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761817.278212:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761817.278217:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761817.278229:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1743 00000100:00080000:0.0:1789761817.278241:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761817.278248:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761817.278313:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504106368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761817.278354:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761817.278358:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761817.278360:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761817.278365:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1743 00000100:00080000:0.0:1789761817.278369:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761817.278374:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761817.278389:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761817.278403:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504106368:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761817.278410:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504106496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761817.278427:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761817.278431:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761817.278433:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761817.278437:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1743 00000100:00080000:0.0:1789761817.278441:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761817.278445:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761817.278451:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761817.278462:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504106624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761817.278474:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761817.278476:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761817.278480:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761817.278483:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761817.278488:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1743 00000100:00080000:0.0:1789761817.278494:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761817.278499:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761817.278506:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761817.278537:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761817/real 1789761817] req@ffff9018c8b98380 x1876699504106368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761817.278547:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504106752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761817.278624:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761817.278627:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761817.278631:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761817.278632:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504106368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761817.278649:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1743 00000100:00080000:0.0:1789761817.278662:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761817.278664:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761817.278670:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789761817.278676:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761817.278680:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504106368:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789761817.278685:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761817.278699:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504106880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761817.278705:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504106496:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761817.278729:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761817.278732:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761817.278734:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761817.278740:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1743 00000800:00000100:2.0:1789761817.278744:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761817.278749:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761817.278756:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761817.278759:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761817.278761:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761817/real 1789761817] req@ffff9018c8b9a300 x1876699504106496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761817.278775:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504107008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761817.278777:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504106496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761817.278834:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761817.278843:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761817.278847:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504106496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761817.278854:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504106624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761817.278873:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761817.278880:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761817/real 1789761817] req@ffff9018c8b9ad80 x1876699504106624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761817.278888:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504106624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761817.278898:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761817.278900:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761817.278902:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504106624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761817.278906:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504106752:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761817.278930:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761817.278940:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761817/real 1789761817] req@ffff9018c8b99500 x1876699504106752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761817.278954:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504106752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761817.278960:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761817.278962:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761817.278964:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504106752:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761817.278976:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504106880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761817.278994:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761817.279005:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761817/real 1789761817] req@ffff9018c8b99c00 x1876699504106880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761817.279019:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504106880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761817.279045:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761817.279072:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761817.279076:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504106880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761817.279082:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504107008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761817.279116:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761817.279124:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761817/real 1789761817] req@ffff9018c8b98380 x1876699504107008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761817.279132:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504107008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761817.279137:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761817.279139:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761817.279141:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504107008:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761817.918179:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761818.942271:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761819.966285:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761820.990191:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761822.014200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761822.398132:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761822.398142:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761822.398147:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761822.398159:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1748 00000100:00080000:3.0:1789761822.398174:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761822.398184:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761822.398233:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504107136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761822.398258:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761822.398261:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761822.398263:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761822.398266:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1748 00000100:00080000:3.0:1789761822.398270:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761822.398273:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761822.398283:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761822.398293:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504107264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761822.398301:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761822.398304:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761822.398305:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761822.398308:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1748 00000100:00080000:3.0:1789761822.398310:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761822.398314:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761822.398320:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761822.398324:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504107136:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761822.398334:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504107392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761822.398347:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761822.398351:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761822.398353:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761822.398358:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1748 00000100:00080000:3.0:1789761822.398363:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761822.398368:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761822.398374:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761822.398390:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504107520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761822.398402:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761822.398406:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761822.398410:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789761822.398412:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761822.398418:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1748 00000100:00080000:3.0:1789761822.398422:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761822.398426:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761822.398431:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761822.398441:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504107648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761822.398448:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761822.398450:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761822.398452:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761822.398456:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1748 00000100:00080000:3.0:1789761822.398459:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761822.398461:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761822.398465:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761822.398473:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761822/real 1789761822] req@ffff9018d070df80 x1876699504107136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761822.398474:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504107776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761822.398513:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504107136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761822.398526:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761822.398531:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761822.398536:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504107136:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761822.398548:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504107264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761822.398584:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761822.398598:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761822/real 1789761822] req@ffff9018d070ed80 x1876699504107264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761822.398612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504107264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761822.398622:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761822.398626:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761822.398629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504107264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761822.398661:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504107392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761822.398683:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761822.398696:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761822/real 1789761822] req@ffff9018d070f480 x1876699504107392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761822.398709:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504107392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761822.398718:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761822.398722:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761822.398726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504107392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761822.398732:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504107520:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761822.398754:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761822.398766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761822/real 1789761822] req@ffff9018d070ca80 x1876699504107520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761822.398780:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504107520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761822.398789:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761822.398792:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761822.398796:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504107520:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761822.398802:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504107648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761822.398824:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761822.398835:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761822/real 1789761822] req@ffff9018d070ce00 x1876699504107648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761822.398858:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504107648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761822.398867:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761822.398870:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761822.398874:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504107648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761822.398894:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504107776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761822.398917:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761822.398929:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761822/real 1789761822] req@ffff9018d070d880 x1876699504107776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761822.398943:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504107776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761822.398957:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761822.398961:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761822.398965:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504107776:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761823.038270:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761824.062230:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761825.086250:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761826.110174:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761827.134256:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761827.518173:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761827.518184:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761827.518188:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761827.518198:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1754 00000100:00080000:3.0:1789761827.518211:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761827.518218:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761827.518271:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504107904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761827.518303:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761827.518307:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761827.518308:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761827.518312:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1754 00000100:00080000:3.0:1789761827.518315:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761827.518319:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761827.518329:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761827.518341:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504108032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761827.518353:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761827.518356:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761827.518357:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761827.518361:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1754 00000100:00080000:3.0:1789761827.518363:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761827.518365:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761827.518369:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761827.518378:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504107904:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761827.518379:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504108160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761827.518401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761827.518403:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761827.518404:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761827.518409:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1754 00000100:00080000:3.0:1789761827.518411:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761827.518415:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761827.518419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761827.518427:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504108288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761827.518435:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761827.518438:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761827.518439:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761827.518442:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1754 00000100:00080000:3.0:1789761827.518448:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761827.518451:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761827.518455:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761827.518463:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504108416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761827.518469:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761827.518472:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761827.518473:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761827.518476:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1754 00000100:00080000:3.0:1789761827.518478:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761827.518483:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761827.518487:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761827.518499:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504108032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789761827.518500:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504108544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761827.518555:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504108160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761827.518574:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504108288:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761827.518589:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504108416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761827.518603:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504108544:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761828.158177:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761829.182180:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761830.206156:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789761830.590475:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789761830.590516:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789761830.590534:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761830.590589:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761830.590605:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761830.590620:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761830.590632:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789761830.590639:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789761830.590730:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761827/real 1789761830] req@ffff9018d070d180 x1876699504108160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761830.590755:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504108160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761830.590768:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761830.590773:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761830.590778:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504108160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761830.590787:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761827/real 1789761830] req@ffff9018d070fb80 x1876699504108288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761830.590801:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504108288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761830.590809:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761830.590813:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761830.590816:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504108288:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761830.590837:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761827/real 1789761830] req@ffff9018d070e680 x1876699504108416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761830.590850:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504108416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761830.590858:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761830.590862:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761830.590866:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504108416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761830.590878:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761827/real 1789761830] req@ffff9018d070e300 x1876699504108544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761830.590894:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504108544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761830.590907:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761830.590911:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761830.590914:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504108544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761830.590919:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761827/real 1789761830] req@ffff9018d070c000 x1876699504107904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761830.590933:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504107904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761830.590941:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761830.590945:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761830.590948:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504107904:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761830.590954:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761827/real 1789761830] req@ffff9018d070dc00 x1876699504108032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761830.590967:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504108032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761830.590974:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761830.590979:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761830.590982:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504108032:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761831.230157:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761832.254184:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761832.638150:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761832.638161:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761832.638165:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761832.638175:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1759 00000100:00080000:0.0:1789761832.638186:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761832.638193:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761832.638228:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504108672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761832.638253:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761832.638256:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761832.638257:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761832.638260:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1759 00000100:00080000:0.0:1789761832.638263:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761832.638266:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761832.638274:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761832.638287:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504108800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761832.638304:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504108672:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761832.638306:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761832.638315:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761832.638318:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761832.638323:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1759 00000100:00080000:0.0:1789761832.638327:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761832.638331:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761832.638337:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761832.638351:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504108928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761832.638360:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761832.638364:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761832.638367:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761832.638371:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1759 00000100:00080000:0.0:1789761832.638375:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789761832.638380:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761832.638384:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761832.638392:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761832.638405:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504109056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761832.638415:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761832.638419:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761832.638420:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761832.638424:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1759 00000100:00080000:0.0:1789761832.638429:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761832.638433:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761832/real 1789761832] req@ffff9018c8b9a680 x1876699504108672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761832.638438:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761832.638452:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761832.638462:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504108672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761832.638468:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504109184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761832.638483:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761832.638485:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761832.638491:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761832.638492:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761832.638496:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761832.638501:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504108672:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761832.638506:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1759 00000100:00080000:0.0:1789761832.638513:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761832.638518:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761832.638524:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761832.638534:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504108800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761832.638556:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504109312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761832.638568:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761832.638589:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761832/real 1789761832] req@ffff9018c8b9b800 x1876699504108800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761832.638604:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504108800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761832.638615:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761832.638619:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761832.638623:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504108800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761832.638630:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504108928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761832.638655:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761832.638668:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761832/real 1789761832] req@ffff9018c8b9aa00 x1876699504108928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761832.638681:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504108928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761832.638690:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761832.638693:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761832.638697:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504108928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761832.638704:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504109056:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761832.638726:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761832.638739:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761832/real 1789761832] req@ffff9018c8b98000 x1876699504109056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761832.638753:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504109056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761832.638762:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761832.638772:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761832.638776:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504109056:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761832.638782:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504109184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761832.638815:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761832.638829:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761832/real 1789761832] req@ffff9018c8b9b480 x1876699504109184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761832.638847:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504109184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761832.638855:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761832.638859:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761832.638863:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504109184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761832.638882:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504109312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761832.638903:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761832.638914:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761832/real 1789761832] req@ffff9018c8b9b100 x1876699504109312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761832.638927:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504109312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761832.638935:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761832.638938:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761832.638941:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504109312:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761833.278156:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761834.302876:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761835.326203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761836.350246:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761837.374177:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761837.758117:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761837.758129:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761837.758133:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761837.758143:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1764 00000100:00080000:0.0:1789761837.758160:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761837.758172:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761837.758212:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504109440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761837.758240:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761837.758245:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761837.758248:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761837.758254:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1764 00000100:00080000:0.0:1789761837.758260:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761837.758265:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761837.758275:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761837.758289:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504109568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761837.758296:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504109440:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761837.758305:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761837.758310:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761837.758313:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761837.758319:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1764 00000100:00080000:0.0:1789761837.758325:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761837.758330:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761837.758336:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761837.758349:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504109696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761837.758361:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789761837.758367:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761837.758368:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761837.758373:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761837.758379:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1764 00000100:00080000:0.0:1789761837.758385:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761837.758390:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761837.758397:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761837.758415:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761837/real 1789761837] req@ffff9018c8b98e00 x1876699504109440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761837.758424:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504109824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761837.758453:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761837.758458:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761837.758460:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761837.758466:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504109440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761837.758467:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1764 00000100:00080000:0.0:1789761837.758474:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761837.758482:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789761837.758484:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789761837.758487:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789761837.758494:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761837.758500:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504109440:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761837.758508:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504109952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761837.758522:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761837.758525:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761837.758528:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761837.758529:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504109568:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789761837.758536:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1764 00000100:00080000:0.0:1789761837.758542:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761837.758546:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761837.758552:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761837.758563:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761837.758568:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504110080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761837.758586:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761837/real 1789761837] req@ffff9018c8b99f80 x1876699504109568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761837.758605:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504109568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761837.758615:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761837.758620:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761837.758625:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504109568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761837.758634:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504109696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761837.758658:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761837.758673:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761837/real 1789761837] req@ffff9018c8b98700 x1876699504109696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761837.758687:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504109696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761837.758702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761837.758706:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761837.758709:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504109696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761837.758716:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504109824:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761837.758737:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761837.758750:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761837/real 1789761837] req@ffff9018c8b9bb80 x1876699504109824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761837.758763:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504109824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761837.758792:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761837.758796:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761837.758800:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504109824:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761837.758814:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504109952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761837.758837:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761837.758850:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761837/real 1789761837] req@ffff9018c8b99880 x1876699504109952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761837.758864:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504109952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761837.758872:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761837.758875:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761837.758879:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504109952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761837.758901:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504110080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761837.758922:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761837.758934:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761837/real 1789761837] req@ffff9018c8b98a80 x1876699504110080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761837.758948:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504110080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761837.758956:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761837.758959:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761837.758963:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504110080:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761838.398154:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761839.422231:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761840.446199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761841.470199:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761842.494248:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761842.878132:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761842.878143:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761842.878147:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761842.878160:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1769 00000100:00080000:0.0:1789761842.878172:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761842.878180:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761842.878223:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504110208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761842.878255:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761842.878260:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761842.878262:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761842.878266:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1769 00000100:00080000:0.0:1789761842.878271:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761842.878275:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761842.878284:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761842.878297:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504110208:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761842.878298:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504110336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761842.878309:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761842.878312:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761842.878314:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761842.878316:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1769 00000100:00080000:0.0:1789761842.878318:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761842.878320:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761842.878323:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761842.878343:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504110464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761842.878350:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761842.878353:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761842.878354:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761842.878358:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1769 00000100:00080000:0.0:1789761842.878363:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789761842.878368:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761842.878373:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761842.878380:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761842.878395:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504110592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761842.878412:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761842.878418:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789761842.878419:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761842/real 1789761842] req@ffff9018c8b98a80 x1876699504110208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761842.878421:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761842.878432:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1769 00000100:00080000:0.0:1789761842.878441:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761842.878447:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761842.878451:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504110208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789761842.878455:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789761842.878471:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789761842.878473:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504110720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761842.878480:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761842.878487:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761842.878490:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504110208:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761842.878491:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761842.878495:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761842.878503:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1769 00000100:00080000:0.0:1789761842.878507:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761842.878510:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761842.878514:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761842.878520:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504110848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761842.878527:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504110336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761842.878555:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761842.878570:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761842/real 1789761842] req@ffff9018c8b99880 x1876699504110336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761842.878584:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504110336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761842.878593:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761842.878595:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761842.878598:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504110336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761842.878602:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504110464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761842.878618:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761842.878626:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761842/real 1789761842] req@ffff9018c8b9bb80 x1876699504110464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761842.878634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504110464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761842.878648:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761842.878651:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761842.878653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504110464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761842.878658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504110592:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761842.878669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761842.878676:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761842/real 1789761842] req@ffff9018c8b98700 x1876699504110592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761842.878688:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504110592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761842.878698:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761842.878701:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761842.878704:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504110592:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761842.878710:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504110720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761842.878745:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761842.878756:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761842/real 1789761842] req@ffff9018c8b99f80 x1876699504110720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761842.878765:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504110720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761842.878770:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761842.878773:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761842.878775:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504110720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761842.878789:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504110848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761842.878808:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761842.878820:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761842/real 1789761842] req@ffff9018c8b98e00 x1876699504110848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761842.878832:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504110848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761842.878849:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761842.878851:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761842.878854:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504110848:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761843.518191:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761844.542180:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761845.566195:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761846.590187:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761847.614240:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761847.998175:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761847.998185:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761847.998190:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761847.998200:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1774 00000100:00080000:0.0:1789761847.998213:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761847.998220:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761847.998265:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504110976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761847.998299:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761847.998304:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761847.998306:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761847.998311:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1774 00000100:00080000:0.0:1789761847.998315:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761847.998319:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761847.998341:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761847.998354:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504111104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761847.998366:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761847.998369:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761847.998371:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761847.998374:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1774 00000100:00080000:0.0:1789761847.998377:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761847.998384:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761847.998388:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504110976:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789761847.998389:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761847.998402:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504111232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761847.998411:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761847.998415:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761847.998416:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761847.998420:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1774 00000100:00080000:0.0:1789761847.998423:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761847.998426:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761847.998430:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761847.998443:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504111360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761847.998451:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761847.998453:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761847.998455:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761847.998457:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1774 00000100:00080000:0.0:1789761847.998459:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761847.998462:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761847.998465:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761847.998476:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504111488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761847.998479:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761847.998490:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761847.998493:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761847.998495:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761847.998498:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1774 00000100:00080000:0.0:1789761847.998501:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761847.998503:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761847.998508:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761847.998515:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504111616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761847.998543:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761847/real 1789761847] req@ffff9018c8b9b100 x1876699504110976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761847.998565:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504110976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761847.998577:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761847.998582:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761847.998586:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504110976:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761847.998595:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504111104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761847.998633:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761847.998647:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761847/real 1789761847] req@ffff9018c8b9b480 x1876699504111104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761847.998660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504111104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761847.998676:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761847.998680:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761847.998683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504111104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761847.998725:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504111232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761847.998748:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761847.998761:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761847/real 1789761847] req@ffff9018c8b98000 x1876699504111232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761847.998772:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504111232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761847.998780:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761847.998784:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761847.998788:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504111232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761847.998795:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504111360:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761847.998815:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761847.998828:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761847/real 1789761847] req@ffff9018c8b9aa00 x1876699504111360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761847.998840:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504111360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761847.998849:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761847.998852:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761847.998856:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504111360:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761847.998863:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504111488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761847.998884:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761847.998895:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761847/real 1789761847] req@ffff9018c8b9b800 x1876699504111488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761847.998906:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504111488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761847.998915:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761847.998919:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761847.998923:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504111488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761847.998929:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504111616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761847.998948:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761847.998958:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761847/real 1789761847] req@ffff9018c8b9a680 x1876699504111616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761847.998970:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504111616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761847.998978:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761847.998982:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761847.998985:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504111616:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761848.638227:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761849.662241:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761850.686212:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761851.710207:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761852.734174:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761853.118184:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761853.118202:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761853.118210:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761853.118222:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1779 00000100:00080000:3.0:1789761853.118234:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761853.118241:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761853.118295:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504111744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761853.118327:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761853.118331:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761853.118333:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761853.118336:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1779 00000100:00080000:3.0:1789761853.118340:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761853.118344:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761853.118355:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761853.118368:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504111744:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761853.118370:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504111872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761853.118382:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761853.118385:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761853.118386:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761853.118389:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1779 00000100:00080000:3.0:1789761853.118391:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761853.118394:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761853.118396:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761853.118405:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504112000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761853.118411:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761853.118413:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761853.118415:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761853.118417:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1779 00000100:00080000:3.0:1789761853.118438:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761853.118440:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761853.118445:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789761853.118449:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761853.118457:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504112128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761853.118466:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761853.118469:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761853.118470:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761853.118474:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1779 00000100:00080000:3.0:1789761853.118476:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761853.118479:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761853.118482:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761853.118491:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504112256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761853.118495:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761853/real 1789761853] req@ffff9018d070f100 x1876699504111744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761853.118509:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761853.118513:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761853.118515:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761853.118519:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1779 00000100:00080000:3.0:1789761853.118521:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761853.118522:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504111744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761853.118526:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761853.118533:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789761853.118539:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761853.118543:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504112384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761853.118550:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761853.118560:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504111744:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761853.118592:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504111872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761853.118615:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761853.118626:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761853/real 1789761853] req@ffff9018d070f800 x1876699504111872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761853.118634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504111872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761853.118640:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761853.118642:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761853.118645:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504111872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761853.118648:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504112000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761853.118678:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761853.118686:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761853/real 1789761853] req@ffff9018d070d500 x1876699504112000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761853.118701:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504112000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761853.118706:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761853.118708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761853.118710:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504112000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761853.118714:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504112128:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761853.118726:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761853.118733:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761853/real 1789761853] req@ffff9018d070dc00 x1876699504112128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761853.118740:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504112128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761853.118745:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761853.118747:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761853.118748:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504112128:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761853.118752:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504112256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761853.118762:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761853.118768:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761853/real 1789761853] req@ffff9018d070c000 x1876699504112256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761853.118775:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504112256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761853.118784:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761853.118786:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761853.118788:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504112256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761853.118791:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504112384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761853.118802:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761853.118808:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761853/real 1789761853] req@ffff9018d070e300 x1876699504112384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761853.118814:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504112384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761853.118819:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761853.118821:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761853.118823:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504112384:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761853.758309:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761854.782303:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761855.806205:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761856.830188:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761857.854201:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761858.238187:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761858.238200:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761858.238207:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761858.238220:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1784 00000100:00080000:3.0:1789761858.238235:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761858.238244:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761858.238299:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504112512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761858.238335:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761858.238340:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761858.238342:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761858.238347:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1784 00000100:00080000:3.0:1789761858.238352:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761858.238357:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761858.238370:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761858.238387:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504112640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761858.238400:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761858.238403:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761858.238406:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761858.238416:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1784 00000100:00100000:2.0:1789761858.238419:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504112512:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761858.238423:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761858.238434:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761858.238441:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761858.238457:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504112768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761858.238475:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761858.238479:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761858.238481:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761858.238486:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1784 00000100:00080000:3.0:1789761858.238492:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789761858.238500:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761858.238500:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761858.238513:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761858.238530:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504112896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761858.238545:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761858.238549:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761858/real 1789761858] req@ffff9018d070e680 x1876699504112512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761858.238555:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761858.238562:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761858.238571:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1784 00000100:00080000:3.0:1789761858.238577:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761858.238581:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504112512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761858.238590:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789761858.238623:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761858.238628:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:3.0:1789761858.238628:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761858.238636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504112512:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761858.238653:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504113024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761858.238666:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504112640:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761858.238671:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761858.238678:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761858.238680:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761858.238687:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1784 00000800:00000100:2.0:1789761858.238694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761858.238696:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761858.238705:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761858.238709:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761858/real 1789761858] req@ffff9018d070fb80 x1876699504112640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789761858.238714:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761858.238725:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504112640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761858.238733:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761858.238738:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504113152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761858.238740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761858.238747:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504112640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761858.238760:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504112768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761858.238792:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761858.238801:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761858/real 1789761858] req@ffff9018d070d180 x1876699504112768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761858.238811:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504112768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761858.238817:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761858.238820:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761858.238822:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504112768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761858.238828:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504112896:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761858.238846:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761858.238853:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761858/real 1789761858] req@ffff9018d070d880 x1876699504112896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761858.238861:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504112896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761858.238867:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761858.238869:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761858.238871:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504112896:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761858.238884:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504113024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761858.238897:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761858.238903:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761858/real 1789761858] req@ffff9018d070ce00 x1876699504113024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761858.238910:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504113024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761858.238916:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761858.238918:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761858.238920:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504113024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761858.238924:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504113152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761858.238934:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761858.238940:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761858/real 1789761858] req@ffff9018d070ca80 x1876699504113152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761858.238948:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504113152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761858.238953:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761858.238955:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761858.238957:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504113152:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761858.878272:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761859.902389:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761860.926182:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761861.950225:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761862.974197:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761863.358170:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761863.358181:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761863.358185:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761863.358196:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1789 00000100:00080000:3.0:1789761863.358207:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761863.358215:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761863.358260:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504113280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761863.358286:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761863.358290:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761863.358292:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761863.358295:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1789 00000100:00080000:3.0:1789761863.358297:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761863.358300:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761863.358309:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761863.358319:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504113408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761863.358329:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761863.358334:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761863.358336:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761863.358341:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1789 00000100:00080000:3.0:1789761863.358344:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761863.358350:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761863.358356:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761863.358367:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504113280:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761863.358373:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504113536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761863.358390:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761863.358395:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761863.358397:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761863.358402:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1789 00000100:00080000:3.0:1789761863.358406:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761863.358409:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761863.358414:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761863.358428:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504113664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761863.358436:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761863.358436:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761863.358445:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761863.358447:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761863.358453:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1789 00000100:00080000:3.0:1789761863.358457:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761863.358461:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761863.358466:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761863.358474:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504113792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761863.358481:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761863.358483:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761863.358484:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761863.358489:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761863/real 1789761863] req@ffff9018d070f480 x1876699504113280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761863.358494:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1789 00000100:00080000:3.0:1789761863.358506:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761863.358517:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761863.358522:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761863.358525:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504113280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761863.358534:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504113920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761863.358539:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761863.358549:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761863.358554:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504113280:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761863.358560:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504113408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761863.358595:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761863.358610:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761863/real 1789761863] req@ffff9018d070ed80 x1876699504113408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761863.358624:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504113408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761863.358630:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761863.358632:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761863.358634:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504113408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761863.358656:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504113536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761863.358675:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761863.358686:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761863/real 1789761863] req@ffff9018d070df80 x1876699504113536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761863.358700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504113536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761863.358709:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761863.358713:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761863.358716:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504113536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761863.358722:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504113664:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761863.358738:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761863.358746:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761863/real 1789761863] req@ffff9018d070ca80 x1876699504113664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761863.358754:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504113664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761863.358764:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761863.358767:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761863.358770:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504113664:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761863.358777:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504113792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761863.358797:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761863.358809:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761863/real 1789761863] req@ffff9018d070ce00 x1876699504113792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761863.358819:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504113792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761863.358825:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761863.358827:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761863.358829:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504113792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761863.358833:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504113920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761863.358848:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761863.358855:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761863/real 1789761863] req@ffff9018d070d880 x1876699504113920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761863.358863:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504113920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761863.358868:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761863.358870:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761863.358872:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504113920:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761863.998310:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761865.022222:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761866.046224:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761867.070241:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761868.094229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761868.478093:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761868.478104:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761868.478108:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761868.478118:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1795 00000100:00080000:3.0:1789761868.478130:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761868.478137:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761868.478177:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504114048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761868.478204:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761868.478207:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761868.478208:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761868.478211:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1795 00000100:00080000:3.0:1789761868.478214:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761868.478217:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761868.478226:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761868.478235:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504114176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761868.478244:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761868.478246:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761868.478248:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761868.478251:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1795 00000100:00080000:3.0:1789761868.478253:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761868.478255:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761868.478259:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761868.478265:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504114304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761868.478274:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761868.478276:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504114048:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761868.478277:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761868.478279:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761868.478286:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1795 00000100:00080000:3.0:1789761868.478289:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761868.478291:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761868.478295:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761868.478302:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504114432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761868.478314:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761868.478317:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761868.478318:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761868.478320:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1795 00000100:00080000:3.0:1789761868.478328:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761868.478331:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761868.478334:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761868.478342:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504114560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761868.478348:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761868.478351:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761868.478352:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761868.478355:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1795 00000100:00080000:3.0:1789761868.478357:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761868.478359:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761868.478362:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761868.478367:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761868.478373:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504114688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761868.478427:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761868/real 1789761868] req@ffff9018d070d180 x1876699504114048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761868.478451:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504114048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761868.478464:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761868.478469:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761868.478474:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504114048:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761868.478483:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504114176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761868.478511:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761868.478527:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761868/real 1789761868] req@ffff9018d070fb80 x1876699504114176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761868.478559:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504114176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761868.478569:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761868.478573:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761868.478577:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504114176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761868.478611:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504114304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761868.478644:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761868.478663:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761868/real 1789761868] req@ffff9018d070e680 x1876699504114304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761868.478678:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504114304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761868.478686:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761868.478689:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761868.478693:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504114304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761868.478700:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504114432:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761868.478724:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761868.478737:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761868/real 1789761868] req@ffff9018d070e300 x1876699504114432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761868.478749:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504114432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761868.478758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761868.478762:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761868.478766:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504114432:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761868.478773:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504114560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761868.478796:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761868.478808:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761868/real 1789761868] req@ffff9018d070c000 x1876699504114560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761868.478822:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504114560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761868.478831:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761868.478834:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761868.478838:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504114560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761868.478844:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504114688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761868.478865:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761868.478877:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761868/real 1789761868] req@ffff9018d070dc00 x1876699504114688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761868.478889:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504114688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761868.478898:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761868.478902:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761868.478905:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504114688:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761869.118194:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761870.142183:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761871.166263:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761872.190197:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761873.214233:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761873.598094:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761873.598104:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761873.598110:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761873.598121:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1800 00000100:00080000:3.0:1789761873.598134:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761873.598141:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761873.598183:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504114816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761873.598219:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761873.598225:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761873.598229:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761873.598237:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1800 00000100:00080000:3.0:1789761873.598243:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761873.598249:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761873.598263:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761873.598266:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504114816:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761873.598286:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504114944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761873.598301:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761873.598305:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761873.598307:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761873.598313:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1800 00000100:00080000:3.0:1789761873.598318:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761873.598323:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761873.598331:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761873.598354:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504115072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761873.598362:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761873.598368:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761873.598373:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761873.598375:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761873.598379:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1800 00000100:00080000:3.0:1789761873.598384:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761873.598388:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761873.598393:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761873.598419:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761873/real 1789761873] req@ffff9018d070d500 x1876699504114816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761873.598442:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504114816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761873.598455:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761873.598461:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761873.598466:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504114816:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789761873.598485:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504115200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761873.598509:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761873.598514:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504114944:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789761873.598515:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761873.598520:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761873.598527:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1800 00000100:00080000:3.0:1789761873.598532:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761873.598538:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761873.598545:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761873.598550:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761873.598562:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504115328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761873.598571:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761873/real 1789761873] req@ffff9018d070f800 x1876699504114944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761873.598592:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504114944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761873.598602:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761873.598605:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761873.598609:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504114944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761873.598616:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504115072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761873.598637:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761873.598650:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761873/real 1789761873] req@ffff9018d070f100 x1876699504115072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761873.598662:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504115072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761873.598671:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761873.598674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761873.598678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504115072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761873.598696:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504115200:192.168.203.158@tcp:8:kworker.0 00000100:00080000:3.0:1789761873.598705:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761873.598709:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761873.598711:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761873.598716:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1800 00000100:00080000:3.0:1789761873.598722:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761873.598724:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761873.598743:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761873/real 1789761873] req@ffff9018d070dc00 x1876699504115200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761873.598761:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504115200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761873.598763:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761873.598773:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761873.598790:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504115456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761873.598796:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761873.598808:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761873.598818:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504115200:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761873.598837:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504115328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761873.598860:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761873.598873:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761873/real 1789761873] req@ffff9018d070c000 x1876699504115328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761873.598886:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504115328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761873.598894:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761873.598898:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761873.598901:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504115328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761873.598907:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504115456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761873.598923:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761873.598932:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761873/real 1789761873] req@ffff9018d070e300 x1876699504115456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761873.598943:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504115456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761873.598952:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761873.598954:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761873.598959:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504115456:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761874.238182:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761875.262271:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761876.286244:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761877.310226:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761878.334188:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761878.718194:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761878.718205:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761878.718209:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761878.718219:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1805 00000100:00080000:1.0:1789761878.718231:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761878.718238:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761878.718284:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504115584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761878.718311:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761878.718314:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761878.718316:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761878.718319:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1805 00000100:00080000:1.0:1789761878.718322:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761878.718325:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761878.718334:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761878.718344:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504115712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761878.718352:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761878.718356:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789761878.718357:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504115584:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761878.718358:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761878.718366:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1805 00000100:00080000:1.0:1789761878.718369:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761878.718371:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761878.718374:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761878.718383:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504115840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761878.718389:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761878.718392:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761878.718393:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761878.718396:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1805 00000100:00080000:1.0:1789761878.718399:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761878.718401:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761878.718405:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761878.718412:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504115968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761878.718420:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761878.718422:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761878.718423:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761878.718426:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1805 00000100:00080000:1.0:1789761878.718428:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761878.718433:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789761878.718434:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:1.0:1789761878.718438:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761878.718447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504116096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761878.718454:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761878.718456:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761878.718457:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761878.718479:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1805 00000100:00080000:1.0:1789761878.718481:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761878.718483:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761878.718486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761878.718497:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504116224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761878.718509:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761878/real 1789761878] req@ffff9018c66a5180 x1876699504115584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761878.718530:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504115584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761878.718543:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761878.718549:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761878.718553:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504115584:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761878.718584:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504115712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761878.718609:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761878.718621:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761878/real 1789761878] req@ffff9018c66a6680 x1876699504115712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761878.718630:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504115712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761878.718636:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761878.718638:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761878.718641:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504115712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761878.718645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504115840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761878.718657:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761878.718664:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761878/real 1789761878] req@ffff9018c66a4700 x1876699504115840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761878.718671:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504115840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761878.718677:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761878.718679:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761878.718681:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504115840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761878.718685:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504115968:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761878.718696:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761878.718701:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761878/real 1789761878] req@ffff9018c66a6300 x1876699504115968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761878.718708:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504115968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761878.718717:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761878.718720:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761878.718723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504115968:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761878.718728:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504116096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761878.718747:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761878.718778:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761878/real 1789761878] req@ffff9018c66a4e00 x1876699504116096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761878.718788:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504116096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761878.718795:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761878.718798:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761878.718800:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504116096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761878.718806:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504116224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761878.718820:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761878.718827:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761878/real 1789761878] req@ffff9018c66a5500 x1876699504116224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761878.718834:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504116224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761878.718840:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761878.718841:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761878.718843:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504116224:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761879.358273:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761880.382279:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761881.406189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761882.430169:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761883.454194:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761883.838117:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761883.838128:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761883.838132:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761883.838140:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1810 00000100:00080000:1.0:1789761883.838150:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761883.838156:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761883.838187:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504116352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761883.838210:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761883.838213:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761883.838214:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761883.838217:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1810 00000100:00080000:1.0:1789761883.838220:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761883.838222:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761883.838230:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761883.838238:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504116480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761883.838247:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761883.838249:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761883.838251:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761883.838253:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1810 00000100:00080000:1.0:1789761883.838255:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761883.838258:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761883.838262:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761883.838269:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504116352:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761883.838270:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504116608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761883.838287:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761883.838289:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761883.838290:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761883.838293:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1810 00000100:00080000:1.0:1789761883.838295:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761883.838297:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761883.838301:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761883.838308:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504116736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761883.838315:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761883.838318:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761883.838320:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761883.838322:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1810 00000100:00080000:1.0:1789761883.838325:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761883.838334:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789761883.838340:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:1.0:1789761883.838341:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761883.838351:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504116864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761883.838359:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761883.838362:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761883.838363:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761883.838366:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1810 00000100:00080000:1.0:1789761883.838370:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761883.838373:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761883.838377:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761883.838381:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761883/real 1789761883] req@ffff9018c66a6d80 x1876699504116352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761883.838388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504116992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761883.838401:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504116352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761883.838410:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761883.838414:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761883.838440:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504116352:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761883.838448:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504116480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761883.838473:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761883.838488:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761883/real 1789761883] req@ffff9018c66a5c00 x1876699504116480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761883.838500:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504116480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761883.838506:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761883.838508:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761883.838511:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504116480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761883.838530:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504116608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761883.838545:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761883.838553:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761883/real 1789761883] req@ffff9018c66a7b80 x1876699504116608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761883.838560:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504116608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761883.838565:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761883.838567:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761883.838569:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504116608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761883.838572:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504116736:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761883.838582:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761883.838589:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761883/real 1789761883] req@ffff9018c66a4000 x1876699504116736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761883.838596:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504116736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761883.838601:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761883.838602:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761883.838604:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504116736:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761883.838607:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504116864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761883.838616:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761883.838622:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761883/real 1789761883] req@ffff9018c66a5f80 x1876699504116864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761883.838628:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504116864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761883.838633:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761883.838635:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761883.838636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504116864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761883.838640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504116992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761883.838649:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761883.838654:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761883/real 1789761883] req@ffff9018c66a7800 x1876699504116992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761883.838661:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504116992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761883.838666:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761883.838668:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761883.838670:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504116992:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761884.478197:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761885.502244:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761886.526205:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761887.550189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761888.574177:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761888.958165:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761888.958175:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761888.958180:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761888.958191:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1815 00000100:00080000:3.0:1789761888.958203:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761888.958211:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761888.958254:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504117120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761888.958284:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761888.958286:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761888.958288:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761888.958291:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1815 00000100:00080000:3.0:1789761888.958294:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761888.958297:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761888.958306:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761888.958316:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504117248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761888.958340:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761888.958346:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789761888.958347:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504117120:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761888.958349:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761888.958360:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1815 00000100:00080000:3.0:1789761888.958365:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761888.958369:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761888.958375:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761888.958404:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504117376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761888.958412:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761888.958416:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761888.958417:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761888.958420:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1815 00000100:00080000:3.0:1789761888.958424:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761888.958426:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761888.958430:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761888.958438:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504117504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761888.958445:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761888.958448:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761888.958449:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761888.958451:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1815 00000800:00000100:0.0:1789761888.958456:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761888.958457:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761888.958463:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761888.958466:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761888.958474:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504117632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761888.958481:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761888.958484:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761888.958486:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761888.958488:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1815 00000100:00080000:3.0:1789761888.958490:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761888.958492:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761888.958495:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761888.958504:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504117760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761888.958510:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761888/real 1789761888] req@ffff9018d070e680 x1876699504117120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761888.958528:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504117120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761888.958538:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761888.958542:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761888.958545:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504117120:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761888.958551:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504117248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761888.958569:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761888.958577:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761888/real 1789761888] req@ffff9018d070fb80 x1876699504117248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761888.958585:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504117248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761888.958591:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761888.958593:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761888.958595:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504117248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761888.958619:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504117376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761888.958631:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761888.958637:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761888/real 1789761888] req@ffff9018d070d180 x1876699504117376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761888.958645:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504117376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761888.958650:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761888.958652:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761888.958654:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504117376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761888.958658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504117504:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761888.958669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761888.958675:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761888/real 1789761888] req@ffff9018d070d880 x1876699504117504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761888.958687:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504117504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761888.958695:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761888.958698:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761888.958701:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504117504:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761888.958709:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504117632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761888.958728:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761888.958740:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761888/real 1789761888] req@ffff9018d070ce00 x1876699504117632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761888.958753:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504117632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761888.958761:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761888.958764:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761888.958767:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504117632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761888.958772:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504117760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761888.958792:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761888.958803:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761888/real 1789761888] req@ffff9018d070ca80 x1876699504117760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761888.958815:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504117760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761888.958824:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761888.958827:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761888.958830:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504117760:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761889.598201:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761890.622189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761891.646226:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761892.670264:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761893.694214:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761894.078158:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761894.078170:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761894.078174:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761894.078184:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1820 00000100:00080000:1.0:1789761894.078196:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761894.078203:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761894.078248:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504117888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761894.078271:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761894.078275:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761894.078276:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761894.078279:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1820 00000100:00080000:1.0:1789761894.078281:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761894.078285:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761894.078295:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761894.078312:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504117888:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761894.078312:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504118016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761894.078330:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761894.078335:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761894.078337:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761894.078362:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1820 00000100:00080000:1.0:1789761894.078366:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761894.078369:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761894.078373:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761894.078384:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504118144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761894.078392:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761894.078394:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761894.078396:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761894.078399:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1820 00000100:00080000:1.0:1789761894.078401:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761894.078404:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761894.078413:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761894.078423:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504118272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761894.078431:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761894.078434:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761894.078435:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761894.078437:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1820 00000100:00080000:1.0:1789761894.078440:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761894.078445:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504118016:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789761894.078445:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761894.078453:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761894.078464:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504118400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761894.078471:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761894.078474:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761894.078476:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761894.078480:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504118144:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789761894.078481:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1820 00000100:00080000:1.0:1789761894.078487:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761894.078491:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761894.078495:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761894.078506:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504118272:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761894.078507:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504118528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761894.078544:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504118400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761894.078568:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504118528:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761894.718290:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761895.742218:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761896.766205:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761897.150258:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:3.0:1789761897.150292:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761897.150305:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761897.150352:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761897.150360:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761897.150366:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761897.150374:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789761897.150382:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789761897.150394:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761894/real 1789761897] req@ffff9018c66a5c00 x1876699504118400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761897.150416:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504118400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761897.150425:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761897.150429:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761897.150433:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504118400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761897.150438:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761894/real 1789761897] req@ffff9018c66a6d80 x1876699504118528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761897.150446:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504118528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761897.150454:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761897.150458:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761897.150462:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504118528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761897.150468:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761894/real 1789761897] req@ffff9018c66a5f80 x1876699504118016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761897.150482:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504118016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761897.150492:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761897.150496:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761897.150499:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504118016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761897.150507:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761894/real 1789761897] req@ffff9018c66a4000 x1876699504118144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761897.150520:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504118144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761897.150528:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761897.150531:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761897.150535:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504118144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761897.150542:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761894/real 1789761897] req@ffff9018c66a7b80 x1876699504118272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761897.150554:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504118272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761897.150562:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761897.150566:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761897.150570:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504118272:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761897.150577:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761894/real 1789761897] req@ffff9018c66a7800 x1876699504117888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761897.150589:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504117888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761897.150608:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761897.150612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761897.150615:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504117888:192.168.203.158@tcp:250:kworker.0 00010000:00010000:3.0:1789761897.790207:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761898.814229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761899.198198:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761899.198212:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761899.198216:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761899.198228:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1825 00000100:00080000:3.0:1789761899.198242:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761899.198251:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761899.198297:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504118656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761899.198345:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761899.198349:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761899.198351:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761899.198356:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1825 00000100:00080000:3.0:1789761899.198360:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761899.198364:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761899.198375:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761899.198389:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504118784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761899.198405:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761899.198408:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761899.198410:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761899.198414:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1825 00000100:00080000:3.0:1789761899.198418:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761899.198421:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761899.198427:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761899.198437:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504118912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761899.198447:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761899.198451:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761899.198453:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761899.198457:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1825 00000100:00080000:3.0:1789761899.198461:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761899.198467:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761899.198482:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761899.198503:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504119040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761899.198509:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504118656:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761899.198518:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761899.198524:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761899.198526:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761899.198531:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1825 00000100:00080000:3.0:1789761899.198535:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761899.198540:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761899.198547:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761899.198562:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504119168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761899.198571:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761899.198575:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761899.198578:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761899.198583:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1825 00000100:00080000:3.0:1789761899.198588:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761899.198593:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761899.198599:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761899.198611:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761899.198615:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504119296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761899.198678:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761899/real 1789761899] req@ffff9018d070df80 x1876699504118656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761899.198701:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504118656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761899.198713:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761899.198718:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761899.198722:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504118656:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761899.198731:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504118784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761899.198772:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761899.198787:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761899/real 1789761899] req@ffff9018d070ed80 x1876699504118784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761899.198800:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504118784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761899.198815:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761899.198819:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761899.198823:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504118784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761899.198830:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504118912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761899.198852:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761899.198865:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761899/real 1789761899] req@ffff9018d070f480 x1876699504118912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761899.198877:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504118912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761899.198885:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761899.198888:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761899.198891:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504118912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761899.198920:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504119040:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761899.198943:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761899.198956:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761899/real 1789761899] req@ffff9018d070ca80 x1876699504119040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761899.198969:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504119040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761899.198978:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761899.198983:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761899.198989:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504119040:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761899.198996:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504119168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761899.199019:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761899.199066:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761899/real 1789761899] req@ffff9018d070ce00 x1876699504119168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761899.199091:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504119168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761899.199100:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761899.199103:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761899.199106:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504119168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761899.199113:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504119296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761899.199135:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761899.199147:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761899/real 1789761899] req@ffff9018d070d880 x1876699504119296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761899.199159:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504119296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761899.199168:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761899.199171:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761899.199175:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504119296:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761899.838154:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761900.862176:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761901.886167:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761902.910175:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761903.934189:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761904.318204:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761904.318216:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761904.318221:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761904.318231:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1830 00000100:00080000:0.0:1789761904.318244:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761904.318251:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761904.318318:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504119424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761904.318351:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761904.318355:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761904.318356:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761904.318360:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1830 00000100:00080000:0.0:1789761904.318363:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761904.318367:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761904.318377:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761904.318390:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504119552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761904.318398:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761904.318401:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761904.318403:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761904.318405:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1830 00000100:00080000:0.0:1789761904.318407:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761904.318411:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761904.318414:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761904.318425:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504119680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761904.318433:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504119424:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761904.318439:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761904.318444:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761904.318445:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761904.318449:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1830 00000100:00080000:0.0:1789761904.318451:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761904.318453:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761904.318458:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761904.318468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504119808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761904.318476:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761904.318479:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761904.318481:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761904.318483:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1830 00000100:00080000:0.0:1789761904.318490:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761904.318493:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761904.318497:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761904.318507:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504119936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761904.318514:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789761904.318526:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761904.318530:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761904.318532:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761904.318538:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1830 00000100:00080000:0.0:1789761904.318544:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761904.318547:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761904.318552:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761904.318562:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504120064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761904.318569:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761904/real 1789761904] req@ffff9018c8b98380 x1876699504119424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761904.318591:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504119424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761904.318602:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761904.318606:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761904.318610:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504119424:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761904.318624:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504119552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761904.318642:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761904.318651:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761904/real 1789761904] req@ffff9018c8b99c00 x1876699504119552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761904.318660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504119552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761904.318667:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761904.318670:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761904.318672:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504119552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761904.318691:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504119680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761904.318703:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761904.318709:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761904/real 1789761904] req@ffff9018c8b99500 x1876699504119680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761904.318716:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504119680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761904.318722:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761904.318724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761904.318726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504119680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761904.318730:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504119808:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761904.318741:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761904.318753:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761904/real 1789761904] req@ffff9018c8b9ad80 x1876699504119808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761904.318765:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504119808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761904.318774:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761904.318778:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761904.318782:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504119808:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761904.318789:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504119936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761904.318809:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761904.318818:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761904/real 1789761904] req@ffff9018c8b9a300 x1876699504119936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761904.318826:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504119936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761904.318832:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761904.318834:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761904.318836:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504119936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761904.318848:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504120064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761904.318860:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761904.318866:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761904/real 1789761904] req@ffff9018c8b9a680 x1876699504120064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761904.318874:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504120064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761904.318879:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761904.318881:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761904.318883:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504120064:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761904.958194:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761905.982228:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761907.006164:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761908.030168:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761909.054167:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761909.438133:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761909.438141:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761909.438146:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761909.438153:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1835 00000100:00080000:1.0:1789761909.438163:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761909.438170:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761909.438201:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504120192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761909.438224:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761909.438239:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761909.438241:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761909.438245:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1835 00000100:00080000:1.0:1789761909.438250:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761909.438254:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761909.438263:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761909.438267:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504120192:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761909.438283:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504120320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761909.438293:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761909.438296:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761909.438297:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761909.438301:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1835 00000100:00080000:1.0:1789761909.438304:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761909.438307:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761909.438311:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761909.438319:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504120448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761909.438327:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761909.438330:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761909.438331:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761909.438336:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761909.438337:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1835 00000100:00080000:1.0:1789761909.438344:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761909.438349:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761909.438354:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761909.438368:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504120576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761909.438386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761909.438387:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761909/real 1789761909] req@ffff9018c66a5500 x1876699504120192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761909.438392:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761909.438397:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761909.438407:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1835 00000100:00080000:1.0:1789761909.438413:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761909.438417:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761909.438423:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504120192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789761909.438423:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761909.438438:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504120704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761909.438440:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761909.438450:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789761909.438450:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761909.438456:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789761909.438458:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504120192:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761909.438459:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761909.438467:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1835 00000100:00080000:1.0:1789761909.438471:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761909.438474:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761909.438477:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761909.438483:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504120832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761909.438491:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504120320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761909.438518:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761909.438529:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761909/real 1789761909] req@ffff9018c66a4e00 x1876699504120320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761909.438538:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504120320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761909.438543:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761909.438546:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761909.438548:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504120320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761909.438552:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504120448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761909.438565:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761909.438571:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761909/real 1789761909] req@ffff9018c66a6300 x1876699504120448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761909.438578:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504120448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761909.438583:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761909.438584:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761909.438586:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504120448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761909.438590:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504120576:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761909.438600:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761909.438606:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761909/real 1789761909] req@ffff9018c66a4700 x1876699504120576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761909.438613:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504120576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761909.438620:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761909.438622:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761909.438626:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504120576:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761909.438632:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504120704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761909.438652:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761909.438675:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761909/real 1789761909] req@ffff9018c66a6680 x1876699504120704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761909.438696:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504120704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761909.438705:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761909.438709:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761909.438713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504120704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761909.438733:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504120832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761909.438755:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761909.438769:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761909/real 1789761909] req@ffff9018c66a5180 x1876699504120832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761909.438782:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504120832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761909.438792:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761909.438796:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761909.438801:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504120832:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761910.078210:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761911.102263:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761912.126326:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761913.150150:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761914.174191:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761914.558141:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761914.558155:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761914.558160:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761914.558172:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1841 00000100:00080000:0.0:1789761914.558185:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761914.558206:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761914.558245:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504120960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761914.558270:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761914.558273:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761914.558275:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761914.558279:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1841 00000100:00080000:0.0:1789761914.558282:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761914.558286:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761914.558296:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761914.558310:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504121088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761914.558311:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504120960:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761914.558334:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761914.558338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761914.558340:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761914.558343:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1841 00000100:00080000:0.0:1789761914.558346:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761914.558348:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761914.558352:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761914.558363:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504121216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761914.558371:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761914.558374:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761914.558376:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761914.558380:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1841 00000100:00080000:0.0:1789761914.558383:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761914.558386:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789761914.558393:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789761914.558394:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761914.558406:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504121344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761914.558415:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761914.558418:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761914.558419:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761914.558422:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1841 00000100:00080000:0.0:1789761914.558424:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761914.558428:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761914.558431:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761914.558440:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504121472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761914.558447:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761914.558453:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789761914.558454:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761914/real 1789761914] req@ffff9018c8b9b800 x1876699504120960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761914.558456:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761914.558464:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1841 00000100:00080000:0.0:1789761914.558471:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761914.558476:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761914.558480:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761914.558488:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504121600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761914.558493:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504120960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761914.558519:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761914.558525:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761914.558530:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504120960:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761914.558554:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504121088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761914.558577:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761914.558593:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761914/real 1789761914] req@ffff9018c8b9aa00 x1876699504121088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761914.558610:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504121088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761914.558623:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761914.558628:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761914.558633:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504121088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761914.558641:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504121216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761914.558668:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761914.558683:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761914/real 1789761914] req@ffff9018c8b98000 x1876699504121216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761914.558700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504121216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761914.558712:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761914.558716:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761914.558721:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504121216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761914.558729:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504121344:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761914.558754:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761914.558777:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761914/real 1789761914] req@ffff9018c8b9b480 x1876699504121344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761914.558788:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504121344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761914.558795:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761914.558798:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761914.558800:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504121344:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761914.558805:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504121472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761914.558826:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761914.558840:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761914/real 1789761914] req@ffff9018c8b9b100 x1876699504121472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761914.558856:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504121472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761914.558876:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761914.558880:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761914.558885:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504121472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761914.558891:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504121600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761914.558909:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761914.558918:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761914/real 1789761914] req@ffff9018c8b98e00 x1876699504121600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761914.558927:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504121600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761914.558934:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761914.558936:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761914.558939:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504121600:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761915.198155:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761916.222291:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761917.246234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761918.270260:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761919.294272:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761919.678220:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761919.678229:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761919.678232:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761919.678241:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1846 00000100:00080000:1.0:1789761919.678251:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761919.678257:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761919.678313:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504121728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761919.678357:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761919.678361:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761919.678363:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761919.678367:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1846 00000100:00080000:1.0:1789761919.678370:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761919.678373:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761919.678382:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761919.678410:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504121856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761919.678418:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761919.678421:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761919.678422:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761919.678426:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1846 00000100:00100000:2.0:1789761919.678428:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504121728:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761919.678429:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761919.678435:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761919.678440:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761919.678447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504121984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761919.678454:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761919.678457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761919.678459:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761919.678461:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1846 00000100:00080000:1.0:1789761919.678464:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761919.678467:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761919.678470:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761919.678479:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504122112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761919.678485:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761919.678490:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1789761919.678490:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761919.678494:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761919.678498:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1846 00000100:00080000:1.0:1789761919.678502:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761919.678505:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761919.678508:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761919.678516:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504122240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761919.678522:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761919.678525:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761919.678526:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761919.678528:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1846 00000100:00080000:1.0:1789761919.678532:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761919.678536:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761919.678537:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761919/real 1789761919] req@ffff9018c66a4380 x1876699504121728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789761919.678541:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761919.678554:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504122368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761919.678561:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504121728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761919.678572:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761919.678576:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761919.678579:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504121728:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761919.678599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504121856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761919.678614:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761919.678623:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761919/real 1789761919] req@ffff9018c66a7100 x1876699504121856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761919.678631:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504121856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761919.678637:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761919.678639:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761919.678642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504121856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761919.678645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504121984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761919.678655:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761919.678660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761919/real 1789761919] req@ffff9018c66a7480 x1876699504121984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761919.678668:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504121984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761919.678673:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761919.678675:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761919.678677:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504121984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761919.678680:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504122112:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761919.678691:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761919.678697:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761919/real 1789761919] req@ffff9018c66a4a80 x1876699504122112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761919.678704:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504122112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761919.678710:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761919.678711:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761919.678713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504122112:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761919.678716:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504122240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761919.678726:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761919.678732:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761919/real 1789761919] req@ffff9018c66a6a00 x1876699504122240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761919.678739:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504122240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761919.678745:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761919.678746:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761919.678748:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504122240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761919.678751:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504122368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761919.678761:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761919.678766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761919/real 1789761919] req@ffff9018c66a5880 x1876699504122368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761919.678778:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504122368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761919.678783:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761919.678785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761919.678792:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504122368:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761920.318246:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761921.342210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761922.366233:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761923.390182:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761924.414216:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761924.798168:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761924.798179:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761924.798184:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761924.798195:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1851 00000100:00080000:1.0:1789761924.798207:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761924.798214:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761924.798259:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504122496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761924.798289:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761924.798293:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761924.798294:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761924.798297:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1851 00000100:00080000:1.0:1789761924.798299:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761924.798302:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761924.798311:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761924.798322:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504122624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761924.798330:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761924.798333:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761924.798334:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761924.798337:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1851 00000100:00080000:1.0:1789761924.798340:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761924.798346:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761924.798346:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504122496:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789761924.798352:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761924.798360:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504122752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761924.798366:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761924.798369:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761924.798370:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761924.798373:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1851 00000100:00080000:1.0:1789761924.798375:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761924.798378:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761924.798381:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761924.798391:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504122880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761924.798397:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761924.798400:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761924.798402:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761924.798404:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1851 00000100:00080000:1.0:1789761924.798407:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761924.798409:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761924.798412:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761924.798421:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504123008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761924.798428:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761924.798437:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761924.798440:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761924.798441:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761924.798446:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1851 00000100:00080000:1.0:1789761924.798448:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761924.798452:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761924.798455:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761924.798464:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504123136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761924.798484:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761924/real 1789761924] req@ffff9018c66a5880 x1876699504122496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761924.798502:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504122496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761924.798511:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761924.798515:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761924.798519:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504122496:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761924.798525:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504122624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761924.798542:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761924.798551:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761924/real 1789761924] req@ffff9018c66a6a00 x1876699504122624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761924.798560:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504122624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761924.798566:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761924.798568:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761924.798570:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504122624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761924.798597:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504122752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761924.798608:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761924.798615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761924/real 1789761924] req@ffff9018c66a4a80 x1876699504122752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761924.798627:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504122752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761924.798635:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761924.798639:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761924.798642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504122752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761924.798649:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504122880:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761924.798670:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761924.798682:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761924/real 1789761924] req@ffff9018c66a7480 x1876699504122880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761924.798690:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504122880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761924.798695:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761924.798697:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761924.798700:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504122880:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761924.798704:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504123008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761924.798717:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761924.798724:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761924/real 1789761924] req@ffff9018c66a7100 x1876699504123008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761924.798731:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504123008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761924.798736:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761924.798738:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761924.798739:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504123008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761924.798743:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504123136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761924.798754:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761924.798759:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761924/real 1789761924] req@ffff9018c66a4380 x1876699504123136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761924.798766:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504123136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761924.798771:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761924.798773:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761924.798775:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504123136:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761925.438234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761926.462277:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761927.486298:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761928.510274:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761929.534249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761929.918169:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761929.918178:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761929.918182:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761929.918190:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1856 00000100:00080000:0.0:1789761929.918200:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761929.918206:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761929.918241:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504123264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761929.918275:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761929.918279:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761929.918281:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761929.918285:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1856 00000100:00080000:0.0:1789761929.918290:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761929.918295:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761929.918305:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761929.918324:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504123392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761929.918332:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504123264:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761929.918342:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761929.918347:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761929.918350:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761929.918355:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1856 00000100:00080000:0.0:1789761929.918359:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761929.918363:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761929.918370:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761929.918382:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504123520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761929.918391:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761929.918395:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761929.918398:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761929.918402:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1856 00000100:00080000:0.0:1789761929.918406:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761929.918410:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761929.918416:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789761929.918426:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761929.918432:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504123648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761929.918446:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761929.918450:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761929.918452:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761929.918458:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1856 00000100:00080000:0.0:1789761929.918463:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761929.918468:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761929.918474:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761929.918482:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761929/real 1789761929] req@ffff9018c8b99f80 x1876699504123264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761929.918493:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504123776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761929.918504:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761929.918507:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504123264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761929.918509:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761929.918515:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761929.918535:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1856 00000100:00080000:0.0:1789761929.918540:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761929.918540:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761929.918551:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789761929.918552:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761929.918563:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504123264:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789761929.918564:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761929.918592:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504123904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761929.918605:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504123392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761929.918634:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761929.918647:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761929/real 1789761929] req@ffff9018c8b98700 x1876699504123392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761929.918655:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504123392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761929.918661:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761929.918664:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761929.918666:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504123392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761929.918671:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504123520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761929.918683:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761929.918689:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761929/real 1789761929] req@ffff9018c8b9bb80 x1876699504123520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761929.918701:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504123520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761929.918706:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761929.918708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761929.918710:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504123520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761929.918714:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504123648:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761929.918725:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761929.918732:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761929/real 1789761929] req@ffff9018c8b99880 x1876699504123648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761929.918740:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504123648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761929.918745:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761929.918747:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761929.918748:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504123648:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761929.918752:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504123776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761929.918770:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761929.918777:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761929/real 1789761929] req@ffff9018c8b98a80 x1876699504123776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761929.918784:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504123776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761929.918794:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761929.918796:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761929.918798:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504123776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761929.918812:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504123904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761929.918823:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761929.918829:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761929/real 1789761929] req@ffff9018c8b98e00 x1876699504123904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761929.918836:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504123904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761929.918841:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761929.918843:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761929.918845:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504123904:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761930.558208:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761931.582278:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761932.606210:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761933.630235:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761934.654204:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761935.039103:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761935.039114:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761935.039118:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761935.039129:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1861 00000100:00080000:0.0:1789761935.039140:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761935.039148:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761935.039192:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504124032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761935.039218:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761935.039221:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761935.039223:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761935.039226:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1861 00000100:00080000:0.0:1789761935.039229:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761935.039233:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761935.039243:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761935.039254:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504124160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761935.039263:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761935.039264:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504124032:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789761935.039267:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761935.039271:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761935.039275:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1861 00000100:00080000:0.0:1789761935.039277:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761935.039279:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761935.039282:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761935.039291:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504124288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761935.039297:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761935.039300:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761935.039302:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761935.039305:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1861 00000100:00080000:0.0:1789761935.039307:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761935.039309:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761935.039313:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789761935.039320:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504124416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761935.039332:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761935.039334:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761935.039336:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761935.039338:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1861 00000100:00080000:0.0:1789761935.039340:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761935.039343:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761935.039346:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761935.039351:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761935.039354:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504124544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761935.039362:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761935.039365:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789761935.039366:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761935.039369:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1861 00000100:00080000:0.0:1789761935.039372:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761935.039374:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789761935.039377:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761935.039383:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504124672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761935.039423:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761935/real 1789761935] req@ffff9018c8b9b100 x1876699504124032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761935.039460:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504124032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761935.039475:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761935.039481:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761935.039486:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504124032:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761935.039516:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504124160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761935.039542:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761935.039556:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761935/real 1789761935] req@ffff9018c8b9b480 x1876699504124160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761935.039570:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504124160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761935.039579:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761935.039582:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761935.039586:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504124160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761935.039592:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504124288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761935.039613:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761935.039625:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761935/real 1789761935] req@ffff9018c8b98000 x1876699504124288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761935.039638:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504124288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761935.039647:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761935.039650:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761935.039653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504124288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761935.039660:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504124416:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761935.039681:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761935.039694:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761935/real 1789761935] req@ffff9018c8b9aa00 x1876699504124416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761935.039707:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504124416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761935.039716:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761935.039719:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761935.039723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504124416:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761935.039729:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504124544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761935.039750:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761935.039761:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761935/real 1789761935] req@ffff9018c8b9b800 x1876699504124544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761935.039774:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504124544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761935.039783:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761935.039786:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761935.039800:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504124544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761935.039806:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504124672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761935.039827:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761935.039839:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761935/real 1789761935] req@ffff9018c8b9a680 x1876699504124672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761935.039860:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504124672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761935.039868:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761935.039883:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761935.039886:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504124672:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761935.678233:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761936.702257:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761937.726238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761938.750221:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761939.774243:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761940.158171:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761940.158183:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761940.158188:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761940.158202:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1866 00000100:00080000:1.0:1789761940.158215:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761940.158223:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761940.158265:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504124800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761940.158299:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761940.158303:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761940.158305:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761940.158310:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1866 00000100:00080000:1.0:1789761940.158315:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761940.158319:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761940.158334:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761940.158343:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504124800:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761940.158350:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504124928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761940.158370:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761940.158374:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761940.158376:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761940.158380:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1866 00000100:00080000:1.0:1789761940.158385:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761940.158390:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761940.158395:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761940.158409:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504125056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761940.158416:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761940.158423:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761940.158427:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761940.158429:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761940.158433:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1866 00000100:00080000:1.0:1789761940.158437:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761940.158440:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761940.158444:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761940.158453:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504125184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761940.158461:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761940.158462:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761940/real 1789761940] req@ffff9018c66a5180 x1876699504124800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761940.158466:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761940.158470:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761940.158475:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1866 00000100:00080000:1.0:1789761940.158477:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761940.158479:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761940.158486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761940.158497:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504125312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761940.158500:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504124800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761940.158515:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761940.158535:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761940.158537:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761940.158538:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761940.158542:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1866 00000100:00080000:2.0:1789761940.158544:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789761940.158546:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761940.158551:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504124800:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761940.158554:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761940.158561:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761940.158571:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504125440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761940.158584:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504124928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761940.158605:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761940.158615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761940/real 1789761940] req@ffff9018c66a6680 x1876699504124928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761940.158624:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504124928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761940.158631:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761940.158634:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761940.158636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504124928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761940.158640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504125056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761940.158652:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761940.158658:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761940/real 1789761940] req@ffff9018c66a4700 x1876699504125056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761940.158666:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504125056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761940.158671:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761940.158673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761940.158675:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504125056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761940.158678:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504125184:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789761940.158689:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761940.158695:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761940/real 1789761940] req@ffff9018c66a6300 x1876699504125184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761940.158702:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504125184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761940.158708:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761940.158710:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761940.158712:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504125184:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789761940.158716:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504125312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761940.158726:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761940.158732:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761940/real 1789761940] req@ffff9018c66a4e00 x1876699504125312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761940.158739:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504125312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761940.158745:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761940.158747:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761940.158748:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504125312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761940.158752:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504125440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789761940.158770:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761940.158776:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761940/real 1789761940] req@ffff9018c66a5500 x1876699504125440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761940.158783:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504125440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761940.158788:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761940.158790:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761940.158792:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504125440:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761940.798214:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761941.822224:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761942.846329:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761943.870266:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761944.894175:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761945.278167:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761945.278178:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761945.278183:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761945.278193:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1871 00000100:00080000:1.0:1789761945.278206:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761945.278215:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761945.278263:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504125568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761945.278305:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761945.278309:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761945.278311:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761945.278317:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1871 00000100:00080000:1.0:1789761945.278321:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761945.278327:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761945.278338:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761945.278346:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504125568:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761945.278355:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504125696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761945.278368:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761945.278372:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761945.278374:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761945.278378:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1871 00000100:00080000:1.0:1789761945.278382:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761945.278386:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761945.278391:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761945.278402:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504125824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761945.278413:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761945.278417:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761945.278419:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761945.278424:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761945.278429:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1871 00000100:00080000:1.0:1789761945.278435:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761945.278440:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761945.278446:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761945.278460:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504125952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761945.278469:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761945/real 1789761945] req@ffff9018c66a7800 x1876699504125568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761945.278474:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761945.278483:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761945.278485:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761945.278490:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1871 00000100:00100000:0.0:1789761945.278492:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504125568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761945.278496:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761945.278505:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789761945.278507:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1789761945.278512:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761945.278519:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761945.278525:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504125568:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761945.278528:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504126080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761945.278540:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761945.278544:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761945.278547:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761945.278551:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504125696:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789761945.278554:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1871 00000100:00080000:1.0:1789761945.278562:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761945.278567:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761945.278574:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761945.278579:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761945.278590:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504126208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761945.278592:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761945/real 1789761945] req@ffff9018c66a7b80 x1876699504125696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761945.278619:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504125696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761945.278639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761945.278643:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761945.278646:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504125696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761945.278651:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504125824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761945.278664:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761945.278672:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761945/real 1789761945] req@ffff9018c66a4000 x1876699504125824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761945.278686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504125824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761945.278692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761945.278694:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761945.278695:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504125824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761945.278699:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504125952:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761945.278711:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761945.278717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761945/real 1789761945] req@ffff9018c66a5f80 x1876699504125952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761945.278725:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504125952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761945.278731:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761945.278733:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761945.278734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504125952:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761945.278738:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504126080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761945.278748:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761945.278755:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761945/real 1789761945] req@ffff9018c66a6d80 x1876699504126080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761945.278763:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504126080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761945.278773:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761945.278775:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761945.278777:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504126080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761945.278790:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504126208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761945.278803:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761945.278809:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761945/real 1789761945] req@ffff9018c66a5c00 x1876699504126208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761945.278817:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504126208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761945.278823:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761945.278825:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761945.278828:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504126208:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761945.918222:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761946.942206:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761947.966210:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761948.990257:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761950.014176:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761950.398139:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761950.398146:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761950.398150:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761950.398168:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1876 00000100:00080000:2.0:1789761950.398177:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761950.398183:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761950.398252:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504126336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761950.398275:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761950.398280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761950.398282:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761950.398286:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1876 00000100:00080000:2.0:1789761950.398291:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761950.398295:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761950.398321:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761950.398334:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504126464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761950.398344:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504126336:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761950.398344:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761950.398347:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761950.398349:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761950.398352:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1876 00000100:00080000:2.0:1789761950.398356:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761950.398359:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761950.398364:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761950.398373:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504126592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761950.398381:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761950.398384:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761950.398385:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761950.398389:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1876 00000100:00080000:2.0:1789761950.398399:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761950.398403:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789761950.398405:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789761950.398408:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761950.398417:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504126720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761950.398426:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761950.398429:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761950.398431:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761950.398434:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1876 00000100:00080000:2.0:1789761950.398437:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761950.398441:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761950.398446:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761950.398449:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761950/real 1789761950] req@ffff9018d0f25500 x1876699504126336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761950.398455:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504126848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761950.398466:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761950.398469:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761950.398471:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761950.398475:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1876 00000100:00100000:0.0:1789761950.398477:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504126336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761950.398478:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761950.398482:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761950.398486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761950.398488:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761950.398492:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789761950.398495:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504126976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761950.398497:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504126336:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761950.398520:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504126464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761950.398544:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761950.398557:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761950/real 1789761950] req@ffff9018d0f24a80 x1876699504126464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761950.398566:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504126464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761950.398571:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761950.398573:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761950.398576:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504126464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761950.398580:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504126592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761950.398592:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761950.398599:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761950/real 1789761950] req@ffff9018d0f25180 x1876699504126592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761950.398606:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504126592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761950.398611:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761950.398612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761950.398614:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504126592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761950.398618:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504126720:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761950.398639:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761950.398646:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761950/real 1789761950] req@ffff9018d0f27800 x1876699504126720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761950.398653:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504126720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761950.398659:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761950.398661:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761950.398663:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504126720:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761950.398666:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504126848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761950.398676:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761950.398683:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761950/real 1789761950] req@ffff9018d0f24380 x1876699504126848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761950.398695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504126848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761950.398702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761950.398706:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761950.398709:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504126848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761950.398715:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504126976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761950.398733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761950.398745:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761950/real 1789761950] req@ffff9018d0f26d80 x1876699504126976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761950.398754:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504126976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761950.398759:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761950.398761:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761950.398771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504126976:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761951.038209:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761952.062223:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761953.086194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761954.110171:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761955.134194:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761955.518137:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761955.518146:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761955.518149:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761955.518157:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1882 00000100:00080000:3.0:1789761955.518166:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761955.518171:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761955.518200:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504127104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761955.518220:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761955.518223:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761955.518224:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761955.518227:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1882 00000100:00080000:3.0:1789761955.518229:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761955.518232:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761955.518257:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761955.518267:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504127232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761955.518275:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761955.518277:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761955.518279:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761955.518284:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1882 00000100:00100000:0.0:1789761955.518286:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504127104:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761955.518287:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761955.518291:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761955.518297:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761955.518323:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504127360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761955.518329:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761955.518332:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761955.518334:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761955.518336:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1882 00000100:00080000:3.0:1789761955.518339:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761955.518342:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761955.518345:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761955.518352:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504127488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761955.518358:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761955.518360:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761955.518362:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761955.518365:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1882 00000100:00080000:3.0:1789761955.518368:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761955.518370:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789761955.518374:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761955.518378:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761955.518385:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504127616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761955.518392:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761955.518394:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761955.518396:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761955.518398:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1882 00000100:00080000:3.0:1789761955.518405:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761955.518408:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761955.518412:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761955/real 1789761955] req@ffff9018d070d180 x1876699504127104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789761955.518414:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761955.518427:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504127744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761955.518433:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504127104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761955.518451:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761955.518455:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761955.518458:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504127104:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761955.518476:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504127232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761955.518492:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761955.518500:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761955/real 1789761955] req@ffff9018d070fb80 x1876699504127232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761955.518508:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504127232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761955.518514:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761955.518516:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761955.518518:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504127232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761955.518522:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504127360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761955.518533:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761955.518538:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761955/real 1789761955] req@ffff9018d070e680 x1876699504127360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761955.518545:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504127360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761955.518550:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761955.518552:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761955.518554:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504127360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761955.518557:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504127488:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761955.518567:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761955.518572:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761955/real 1789761955] req@ffff9018d070e300 x1876699504127488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761955.518579:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504127488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761955.518584:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761955.518586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761955.518588:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504127488:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761955.518591:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504127616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761955.518601:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761955.518606:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761955/real 1789761955] req@ffff9018d070c000 x1876699504127616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761955.518612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504127616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761955.518617:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761955.518619:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761955.518621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504127616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761955.518624:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504127744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761955.518633:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761955.518639:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761955/real 1789761955] req@ffff9018d070dc00 x1876699504127744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761955.518646:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504127744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761955.518651:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761955.518652:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761955.518654:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504127744:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761956.158180:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761957.182234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761958.206288:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761959.230272:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761960.254161:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761960.638152:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761960.638159:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761960.638163:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761960.638172:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1887 00000100:00080000:2.0:1789761960.638180:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761960.638186:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761960.638223:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504127872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761960.638242:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761960.638244:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761960.638245:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761960.638248:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1887 00000100:00080000:2.0:1789761960.638251:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761960.638254:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761960.638261:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761960.638270:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504128000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761960.638277:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761960.638279:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761960.638281:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761960.638283:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1887 00000100:00080000:2.0:1789761960.638285:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761960.638287:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761960.638290:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761960.638295:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504127872:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761960.638299:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504128128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761960.638307:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761960.638309:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761960.638310:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761960.638312:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1887 00000100:00080000:2.0:1789761960.638315:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761960.638317:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761960.638321:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761960.638331:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504128256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761960.638342:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761960.638345:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761960.638347:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761960.638352:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1887 00000100:00080000:2.0:1789761960.638357:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761960.638362:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761960.638368:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761960.638381:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504128384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761960.638388:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504128000:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789761960.638390:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761960.638398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761960.638400:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761960.638404:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1887 00000100:00080000:2.0:1789761960.638408:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761960.638411:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761960.638415:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761960.638425:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504128512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761960.638431:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504128128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761960.638451:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504128256:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761960.638467:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504128384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761960.638485:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504128512:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761961.278230:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761962.302264:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761963.326193:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761963.710282:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:1.0:1789761963.710319:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789761963.710336:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789761963.710381:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789761963.710389:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789761963.710394:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789761963.710400:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789761963.710409:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789761963.710430:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761960/real 1789761963] req@ffff9018d0f25c00 x1876699504128128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761963.710451:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504128128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761963.710459:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761963.710463:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761963.710467:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504128128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761963.710474:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761960/real 1789761963] req@ffff9018d0f26a00 x1876699504128256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761963.710487:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504128256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761963.710496:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761963.710500:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761963.710514:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504128256:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761963.710522:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761960/real 1789761963] req@ffff9018d0f26300 x1876699504128384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761963.710535:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504128384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761963.710544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761963.710547:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761963.710551:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504128384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761963.710558:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761960/real 1789761963] req@ffff9018d0f27100 x1876699504127872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761963.710572:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504127872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761963.710580:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761963.710584:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761963.710587:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504127872:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761963.710594:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761960/real 1789761963] req@ffff9018d0f25f80 x1876699504128000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761963.710607:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504128000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761963.710615:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761963.710618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761963.710621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504128000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761963.710657:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761960/real 1789761963] req@ffff9018d0f26d80 x1876699504128512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761963.710675:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504128512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761963.710684:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761963.710688:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761963.710691:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504128512:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761964.350180:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761965.374192:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761965.758207:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761965.758218:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761965.758222:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761965.758232:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1892 00000100:00080000:1.0:1789761965.758248:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761965.758255:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761965.758288:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504128640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761965.758317:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761965.758321:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761965.758323:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761965.758327:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1892 00000100:00080000:1.0:1789761965.758331:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761965.758336:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761965.758346:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761965.758360:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504128768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761965.758370:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504128640:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761965.758375:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761965.758381:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761965.758383:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761965.758388:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1892 00000100:00080000:1.0:1789761965.758392:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761965.758397:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761965.758402:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761965.758422:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504128896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761965.758433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761965.758437:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761965.758439:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761965.758445:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1892 00000800:00000100:0.0:1789761965.758447:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761965.758457:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761965.758462:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761965.758467:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761965.758479:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504129024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761965.758491:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761965.758496:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761965.758498:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761965.758502:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1892 00000100:00080000:1.0:1789761965.758507:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761965.758511:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761965.758517:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761965.758522:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761965/real 1789761965] req@ffff9018c66a5c00 x1876699504128640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761965.758533:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504129152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761965.758547:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761965.758551:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504128640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761965.758552:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761965.758556:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761965.758564:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1892 00000100:00080000:1.0:1789761965.758570:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761965.758575:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761965.758576:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761965.758586:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761965.758587:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761965.758596:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504128640:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761965.758602:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504129280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761965.758625:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504128768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761965.758652:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761965.758663:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761965/real 1789761965] req@ffff9018c66a6d80 x1876699504128768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761965.758676:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504128768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761965.758691:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761965.758695:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761965.758699:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504128768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761965.758706:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504128896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761965.758726:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761965.758737:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761965/real 1789761965] req@ffff9018c66a5f80 x1876699504128896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761965.758749:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504128896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761965.758757:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761965.758761:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761965.758765:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504128896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761965.758771:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504129024:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761965.758793:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761965.758822:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761965/real 1789761965] req@ffff9018c66a4000 x1876699504129024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761965.758837:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504129024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761965.758846:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761965.758850:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761965.758854:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504129024:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761965.758860:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504129152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761965.758880:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761965.758894:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761965/real 1789761965] req@ffff9018c66a7b80 x1876699504129152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761965.758907:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504129152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761965.758916:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761965.758919:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761965.758922:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504129152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761965.758927:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504129280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761965.758947:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761965.758958:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761965/real 1789761965] req@ffff9018c66a7800 x1876699504129280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761965.758970:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504129280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761965.758978:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761965.758981:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761965.758984:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504129280:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761966.398151:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761967.422214:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761968.446255:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761969.470199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761970.494213:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761970.878166:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761970.878178:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761970.878182:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761970.878193:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1897 00000100:00080000:2.0:1789761970.878206:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761970.878213:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761970.878265:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504129408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761970.878298:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761970.878303:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761970.878305:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761970.878310:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1897 00000100:00080000:2.0:1789761970.878314:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761970.878319:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761970.878331:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761970.878346:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504129536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761970.878358:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761970.878364:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504129408:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761970.878365:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761970.878375:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761970.878383:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1897 00000100:00080000:2.0:1789761970.878389:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761970.878395:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761970.878401:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761970.878418:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504129664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761970.878425:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761970.878428:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761970.878430:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761970.878434:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1897 00000100:00080000:2.0:1789761970.878438:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761970.878444:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789761970.878445:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789761970.878458:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761970.878469:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504129792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761970.878477:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761970.878479:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761970.878481:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761970.878484:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1897 00000100:00080000:2.0:1789761970.878488:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761970.878493:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789761970.878497:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761970/real 1789761970] req@ffff9018d0f24380 x1876699504129408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761970.878500:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761970.878523:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504129408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761970.878531:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504129920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761970.878542:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761970.878548:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761970.878554:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789761970.878556:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761970.878560:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761970.878561:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504129408:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761970.878572:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1897 00000100:00080000:2.0:1789761970.878575:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761970.878579:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761970.878582:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761970.878593:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504129536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789761970.878594:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504130048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761970.878633:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761970.878646:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761970/real 1789761970] req@ffff9018d0f27800 x1876699504129536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761970.878655:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504129536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761970.878661:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761970.878664:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761970.878666:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504129536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761970.878671:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504129664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761970.878682:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761970.878688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761970/real 1789761970] req@ffff9018d0f25180 x1876699504129664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761970.878701:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504129664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761970.878706:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761970.878708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761970.878710:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504129664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761970.878714:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504129792:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761970.878728:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761970.878738:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761970/real 1789761970] req@ffff9018d0f24a80 x1876699504129792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761970.878751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504129792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761970.878760:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761970.878764:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761970.878767:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504129792:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761970.878787:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504129920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761970.878804:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761970.878813:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761970/real 1789761970] req@ffff9018d0f25500 x1876699504129920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761970.878820:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504129920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761970.878830:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761970.878832:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761970.878834:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504129920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761970.878847:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504130048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761970.878858:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761970.878864:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761970/real 1789761970] req@ffff9018d0f27480 x1876699504130048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761970.878871:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504130048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761970.878876:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761970.878878:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761970.878880:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504130048:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761971.518159:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761972.542204:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761973.566189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761974.590235:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761975.614227:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761975.998187:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761975.998195:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761975.998198:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761975.998206:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1902 00000100:00080000:1.0:1789761975.998216:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761975.998256:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761975.998288:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504130176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761975.998309:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761975.998313:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761975.998314:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761975.998317:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1902 00000100:00080000:1.0:1789761975.998319:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761975.998322:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761975.998329:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761975.998338:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504130304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761975.998345:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761975.998347:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761975.998349:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761975.998352:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1902 00000100:00080000:1.0:1789761975.998353:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761975.998355:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761975.998358:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761975.998364:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504130432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761975.998372:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761975.998374:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761975.998377:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761975.998382:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504130176:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789761975.998383:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1902 00000100:00080000:1.0:1789761975.998388:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761975.998394:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761975.998398:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761975.998407:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504130560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761975.998413:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761975.998415:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761975.998416:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761975.998419:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1902 00000100:00080000:1.0:1789761975.998422:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761975.998425:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761975.998427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761975.998434:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504130688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761975.998440:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761975.998443:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761975.998444:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761975.998446:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1902 00000100:00080000:1.0:1789761975.998449:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761975.998454:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789761975.998455:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:1.0:1789761975.998458:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761975.998468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504130816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761975.998508:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761975/real 1789761975] req@ffff9018c66a5500 x1876699504130176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761975.998529:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504130176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761975.998541:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761975.998547:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761975.998551:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504130176:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761975.998559:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504130304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761975.998584:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761975.998598:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761975/real 1789761975] req@ffff9018c66a4e00 x1876699504130304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761975.998612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504130304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761975.998622:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761975.998626:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761975.998630:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504130304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761975.998655:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504130432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761975.998676:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761975.998687:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761975/real 1789761975] req@ffff9018c66a6300 x1876699504130432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761975.998701:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504130432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761975.998710:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761975.998714:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761975.998718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504130432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761975.998725:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504130560:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761975.998762:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761975.998774:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761975/real 1789761975] req@ffff9018c66a4700 x1876699504130560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761975.998789:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504130560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761975.998803:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761975.998807:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761975.998811:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504130560:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761975.998818:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504130688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761975.998839:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761975.998851:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761975/real 1789761975] req@ffff9018c66a6680 x1876699504130688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761975.998864:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504130688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761975.998873:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761975.998877:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761975.998880:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504130688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761975.998887:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504130816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761975.998907:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761975.998918:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761975/real 1789761975] req@ffff9018c66a5180 x1876699504130816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761975.998939:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504130816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761975.998948:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761975.998951:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761975.998954:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504130816:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761976.638185:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761977.662208:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761978.686238:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761979.710263:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761980.734267:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761981.118221:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761981.118232:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761981.118236:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761981.118247:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1907 00000100:00080000:2.0:1789761981.118259:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761981.118275:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761981.118316:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504130944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761981.118354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761981.118358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761981.118360:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761981.118365:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1907 00000100:00080000:2.0:1789761981.118370:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761981.118375:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761981.118387:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761981.118416:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504130944:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761981.118416:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504131072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761981.118449:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761981.118452:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761981.118455:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761981.118463:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1907 00000100:00080000:2.0:1789761981.118469:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761981.118475:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761981.118481:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761981.118498:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504131200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761981.118508:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761981.118512:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761981.118514:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761981.118518:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1907 00000100:00080000:2.0:1789761981.118523:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761981.118530:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789761981.118534:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789761981.118539:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789761981.118555:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504131328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761981.118566:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761981.118570:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761981.118572:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761981.118577:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1907 00000100:00080000:2.0:1789761981.118582:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761981.118588:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761981.118594:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761981.118595:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761981/real 1789761981] req@ffff9018d0f24e00 x1876699504130944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761981.118613:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504131456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761981.118626:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761981.118627:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504130944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761981.118639:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761981.118646:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761981.118657:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1907 00000100:00080000:1.0:1789761981.118663:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761981.118664:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761981.118676:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789761981.118678:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:2.0:1789761981.118688:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761981.118709:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504130944:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789761981.118721:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504131584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761981.118914:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504131072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761981.118973:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761981.118993:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761981/real 1789761981] req@ffff9018d0f24000 x1876699504131072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761981.119004:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504131072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761981.119012:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761981.119015:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761981.119017:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504131072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761981.119022:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504131200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761981.119065:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761981.119075:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761981/real 1789761981] req@ffff9018d0f24700 x1876699504131200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761981.119089:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504131200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761981.119100:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761981.119102:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761981.119105:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504131200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761981.119109:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504131328:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789761981.119123:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761981.119129:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761981/real 1789761981] req@ffff9018d0f26680 x1876699504131328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761981.119136:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504131328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761981.119142:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761981.119143:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761981.119145:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504131328:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789761981.119148:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504131456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761981.119158:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761981.119164:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761981/real 1789761981] req@ffff9018d0f27480 x1876699504131456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761981.119171:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504131456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761981.119176:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761981.119178:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761981.119181:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504131456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789761981.119184:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504131584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789761981.119195:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789761981.119201:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761981/real 1789761981] req@ffff9018d0f25500 x1876699504131584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761981.119208:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504131584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761981.119213:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761981.119214:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789761981.119216:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504131584:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761981.758214:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761982.782191:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761983.806165:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761984.830219:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761985.854234:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761986.238202:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761986.238213:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761986.238218:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761986.238229:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1912 00000100:00080000:1.0:1789761986.238242:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761986.238250:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789761986.238302:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504131712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761986.238332:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761986.238335:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761986.238337:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761986.238340:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1912 00000100:00080000:1.0:1789761986.238343:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761986.238348:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761986.238358:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761986.238369:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504131840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761986.238375:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761986.238378:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761986.238379:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761986.238382:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1912 00000100:00080000:1.0:1789761986.238384:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761986.238386:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761986.238390:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761986.238393:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504131712:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789761986.238398:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504131968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761986.238406:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761986.238409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761986.238410:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761986.238413:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1912 00000100:00080000:1.0:1789761986.238417:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761986.238421:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761986.238427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789761986.238439:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504132096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761986.238450:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761986.238454:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761986.238457:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761986.238461:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1912 00000100:00080000:1.0:1789761986.238465:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761986.238471:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789761986.238472:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761986.238480:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761986.238490:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504132224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761986.238497:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761986.238499:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789761986.238501:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761986.238504:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1912 00000100:00080000:1.0:1789761986.238506:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761986.238509:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789761986.238513:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761986.238521:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761986/real 1789761986] req@ffff9018c66a4380 x1876699504131712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761986.238523:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504132352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761986.238559:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504131712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761986.238570:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761986.238574:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761986.238578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504131712:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761986.238585:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504131840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761986.238611:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761986.238620:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761986/real 1789761986] req@ffff9018c66a7100 x1876699504131840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761986.238634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504131840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761986.238639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761986.238642:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761986.238644:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504131840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761986.238687:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504131968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761986.238702:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761986.238709:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761986/real 1789761986] req@ffff9018c66a7480 x1876699504131968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761986.238717:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504131968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761986.238729:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761986.238732:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761986.238734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504131968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761986.238738:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504132096:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761986.238750:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761986.238757:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761986/real 1789761986] req@ffff9018c66a4a80 x1876699504132096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761986.238764:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504132096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761986.238774:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761986.238776:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761986.238778:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504132096:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761986.238782:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504132224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761986.238791:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761986.238797:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761986/real 1789761986] req@ffff9018c66a6a00 x1876699504132224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761986.238804:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504132224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761986.238809:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761986.238811:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761986.238813:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504132224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761986.238816:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504132352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761986.238826:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761986.238831:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761986/real 1789761986] req@ffff9018c66a5880 x1876699504132352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761986.238838:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504132352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761986.238842:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761986.238844:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761986.238847:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504132352:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789761986.878249:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761987.902172:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761988.926177:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761989.950285:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761990.974222:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761991.358156:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761991.358167:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761991.358171:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761991.358182:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1917 00000100:00080000:3.0:1789761991.358195:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761991.358202:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789761991.358246:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504132480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761991.358273:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761991.358276:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761991.358277:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761991.358280:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1917 00000100:00080000:3.0:1789761991.358283:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761991.358287:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761991.358297:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761991.358307:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504132608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761991.358315:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504132480:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789761991.358317:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761991.358324:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761991.358325:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761991.358328:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1917 00000100:00080000:3.0:1789761991.358330:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761991.358332:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761991.358335:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761991.358343:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504132736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761991.358349:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761991.358351:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761991.358352:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761991.358354:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1917 00000100:00080000:3.0:1789761991.358357:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761991.358360:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761991.358364:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789761991.358373:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504132864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761991.358385:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761991.358388:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761991.358390:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761991.358392:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1917 00000100:00080000:3.0:1789761991.358394:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761991.358396:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761991.358399:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761991.358405:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789761991.358410:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504132992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761991.358421:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761991.358425:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789761991.358428:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761991.358432:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1917 00000100:00080000:3.0:1789761991.358437:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761991.358443:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789761991.358448:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761991.358455:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761991/real 1789761991] req@ffff9018d070f100 x1876699504132480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761991.358463:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504133120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761991.358476:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504132480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761991.358486:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761991.358490:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761991.358494:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504132480:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761991.358519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504132608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761991.358537:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761991.358546:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761991/real 1789761991] req@ffff9018d070f800 x1876699504132608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761991.358555:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504132608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761991.358561:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761991.358563:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761991.358565:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504132608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761991.358569:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504132736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761991.358580:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761991.358586:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761991/real 1789761991] req@ffff9018d070d500 x1876699504132736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761991.358594:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504132736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761991.358599:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761991.358601:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761991.358603:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504132736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761991.358606:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504132864:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761991.358640:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761991.358653:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761991/real 1789761991] req@ffff9018d070dc00 x1876699504132864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761991.358666:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504132864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761991.358676:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761991.358679:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761991.358681:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504132864:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761991.358686:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504132992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761991.358702:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761991.358710:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761991/real 1789761991] req@ffff9018d070c000 x1876699504132992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761991.358723:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504132992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761991.358729:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761991.358731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761991.358733:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504132992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761991.358736:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504133120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761991.358746:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761991.358752:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761991/real 1789761991] req@ffff9018d070e300 x1876699504133120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761991.358759:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504133120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761991.358765:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761991.358766:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761991.358768:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504133120:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789761991.998239:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761993.022195:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761994.046217:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761995.070207:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761996.094176:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761996.478175:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761996.478185:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761996.478190:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761996.478202:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1923 00000100:00080000:2.0:1789761996.478215:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761996.478221:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789761996.478278:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504133248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761996.478310:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761996.478315:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761996.478318:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761996.478323:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1923 00000100:00080000:2.0:1789761996.478327:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761996.478332:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761996.478345:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761996.478361:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504133376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761996.478376:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504133248:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789761996.478379:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761996.478396:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761996.478399:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761996.478404:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1923 00000100:00080000:2.0:1789761996.478407:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761996.478412:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761996.478417:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761996.478429:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504133504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761996.478435:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761996.478439:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761996.478441:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761996.478444:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1923 00000100:00080000:2.0:1789761996.478446:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761996.478449:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761996.478455:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789761996.478467:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789761996.478467:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504133632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761996.478485:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761996.478489:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761996.478490:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761996.478493:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1923 00000100:00080000:2.0:1789761996.478496:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761996.478501:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761996.478505:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761996.478512:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504133760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761996.478518:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761996.478521:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789761996.478524:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761996.478527:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1923 00000100:00080000:2.0:1789761996.478530:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761996.478534:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789761996.478538:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761996.478541:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761996/real 1789761996] req@ffff9018d0f24a80 x1876699504133248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761996.478551:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504133888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761996.478591:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504133248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761996.478604:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761996.478610:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761996.478615:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504133248:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789761996.478637:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504133376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761996.478658:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761996.478667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761996/real 1789761996] req@ffff9018d0f25180 x1876699504133376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761996.478675:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504133376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761996.478681:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761996.478683:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761996.478686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504133376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761996.478689:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504133504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761996.478700:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761996.478706:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761996/real 1789761996] req@ffff9018d0f27800 x1876699504133504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761996.478713:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504133504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761996.478718:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761996.478720:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761996.478722:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504133504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761996.478726:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504133632:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789761996.478737:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761996.478743:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761996/real 1789761996] req@ffff9018d0f24380 x1876699504133632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761996.478750:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504133632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761996.478755:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761996.478757:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761996.478759:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504133632:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789761996.478762:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504133760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761996.478772:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761996.478778:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761996/real 1789761996] req@ffff9018d0f26d80 x1876699504133760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761996.478785:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504133760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761996.478789:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761996.478791:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761996.478792:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504133760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789761996.478796:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504133888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789761996.478806:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789761996.478811:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761996/real 1789761996] req@ffff9018d0f25f80 x1876699504133888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761996.478819:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504133888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761996.478824:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761996.478825:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789761996.478827:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504133888:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789761997.118146:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761998.142271:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761999.166190:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762000.190195:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762001.214189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762001.598097:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762001.598105:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762001.598109:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762001.598117:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1928 00000100:00080000:3.0:1789762001.598135:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762001.598141:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762001.598173:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504134016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762001.598192:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762001.598195:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762001.598196:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762001.598199:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1928 00000100:00080000:3.0:1789762001.598201:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762001.598204:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762001.598211:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762001.598219:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504134144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762001.598225:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762001.598227:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762001.598228:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762001.598230:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1928 00000100:00080000:3.0:1789762001.598232:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762001.598234:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762001.598236:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762001.598242:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504134272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762001.598253:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762001.598255:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762001.598257:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762001.598259:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1928 00000100:00080000:3.0:1789762001.598261:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762001.598267:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762001.598269:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504134016:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789762001.598271:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762001.598282:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504134400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762001.598288:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762001.598291:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762001.598292:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762001.598294:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1928 00000100:00080000:3.0:1789762001.598296:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762001.598299:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762001.598302:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762001.598309:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504134528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762001.598315:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762001.598317:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762001.598319:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762001.598322:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1928 00000100:00080000:3.0:1789762001.598325:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762001.598331:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762001.598332:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762001.598340:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762001.598353:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504134656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762001.598366:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762001/real 1789762001] req@ffff9018d070e680 x1876699504134016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762001.598380:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504134016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762001.598388:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762001.598391:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762001.598394:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504134016:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762001.598399:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504134144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762001.598413:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762001.598420:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762001/real 1789762001] req@ffff9018d070fb80 x1876699504134144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762001.598427:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504134144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762001.598433:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762001.598435:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762001.598438:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504134144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762001.598441:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504134272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762001.598450:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762001.598456:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762001/real 1789762001] req@ffff9018d070d180 x1876699504134272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762001.598462:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504134272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762001.598467:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762001.598469:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762001.598471:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504134272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762001.598490:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504134400:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762001.598500:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762001.598506:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762001/real 1789762001] req@ffff9018d070d880 x1876699504134400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762001.598513:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504134400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762001.598518:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762001.598519:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762001.598522:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504134400:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762001.598525:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504134528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762001.598534:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762001.598557:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762001/real 1789762001] req@ffff9018d070ce00 x1876699504134528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762001.598565:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504134528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762001.598571:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762001.598572:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762001.598574:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504134528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762001.598578:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504134656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762001.598591:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762001.598597:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762001/real 1789762001] req@ffff9018d070ca80 x1876699504134656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762001.598604:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504134656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762001.598609:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762001.598611:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762001.598612:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504134656:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762002.238227:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762003.262153:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762004.286237:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762005.310191:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762006.334208:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762006.718150:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762006.718165:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762006.718169:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762006.718181:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1933 00000100:00080000:2.0:1789762006.718195:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762006.718203:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762006.718251:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504134784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762006.718288:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762006.718292:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762006.718294:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762006.718299:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1933 00000100:00080000:2.0:1789762006.718304:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762006.718309:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762006.718321:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762006.718338:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504134912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762006.718349:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762006.718354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762006.718355:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762006.718360:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1933 00000100:00080000:2.0:1789762006.718363:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762006.718373:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504134784:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762006.718373:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762006.718380:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762006.718397:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504135040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762006.718424:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762006.718428:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762006.718430:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762006.718435:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1933 00000100:00080000:2.0:1789762006.718440:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762006.718446:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762006.718452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789762006.718460:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762006.718470:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504135168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762006.718484:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762006.718488:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762006.718490:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762006.718494:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1933 00000100:00080000:2.0:1789762006.718499:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762006.718517:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762006.718525:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762006/real 1789762006] req@ffff9018d0f27100 x1876699504134784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789762006.718526:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762006.718548:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504135296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762006.718557:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504134784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762006.718566:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762006.718577:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762006.718582:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762006.718592:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1933 00000100:00080000:0.0:1789762006.718593:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762006.718598:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762006.718604:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789762006.718609:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762006.718617:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504134784:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789762006.718619:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762006.718638:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504134912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762006.718644:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504135424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762006.718681:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762006.718698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762006/real 1789762006] req@ffff9018d0f26300 x1876699504134912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762006.718713:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504134912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762006.718722:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762006.718726:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762006.718730:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504134912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762006.718756:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504135040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762006.718777:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762006.718789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762006/real 1789762006] req@ffff9018d0f26a00 x1876699504135040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762006.718809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504135040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762006.718817:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762006.718820:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762006.718824:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504135040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762006.718831:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504135168:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762006.718852:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762006.718864:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762006/real 1789762006] req@ffff9018d0f25c00 x1876699504135168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762006.718877:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504135168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762006.718886:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762006.718889:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762006.718893:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504135168:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762006.718900:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504135296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762006.718921:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762006.718933:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762006/real 1789762006] req@ffff9018d0f25f80 x1876699504135296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762006.718944:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504135296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762006.718959:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762006.718971:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762006.718975:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504135296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762006.718982:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504135424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762006.719005:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762006.719017:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762006/real 1789762006] req@ffff9018d0f26d80 x1876699504135424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762006.719030:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504135424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762006.719085:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762006.719089:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762006.719093:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504135424:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762007.358276:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762008.382153:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762009.406221:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762010.430218:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762011.454195:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762011.838226:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762011.838236:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762011.838240:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762011.838250:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1938 00000100:00080000:3.0:1789762011.838263:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762011.838269:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762011.838314:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504135552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762011.838341:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762011.838345:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762011.838346:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762011.838349:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1938 00000100:00080000:3.0:1789762011.838352:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762011.838354:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762011.838364:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762011.838374:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504135680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762011.838383:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762011.838386:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762011.838388:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762011.838391:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1938 00000100:00080000:3.0:1789762011.838393:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762011.838396:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762011.838399:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762011.838407:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504135808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762011.838416:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762011.838419:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762011.838420:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762011.838424:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1938 00000100:00080000:3.0:1789762011.838429:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762011.838431:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504135552:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762011.838434:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762011.838449:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762011.838477:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504135936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762011.838486:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762011.838489:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762011.838490:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762011.838493:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1938 00000100:00080000:3.0:1789762011.838498:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762011.838502:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762011.838507:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762011.838517:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504136064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762011.838524:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762011.838537:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762011.838541:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762011.838542:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762011.838547:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1938 00000100:00080000:3.0:1789762011.838554:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762011.838558:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762011.838564:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762011.838582:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504136192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762011.838587:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762011/real 1789762011] req@ffff9018d070f480 x1876699504135552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762011.838636:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504135552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762011.838645:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762011.838649:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762011.838655:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504135552:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762011.838663:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504135680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762011.838680:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762011.838689:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762011/real 1789762011] req@ffff9018d070ed80 x1876699504135680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762011.838697:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504135680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762011.838702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762011.838705:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762011.838707:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504135680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762011.838727:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504135808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762011.838738:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762011.838744:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762011/real 1789762011] req@ffff9018d070df80 x1876699504135808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762011.838751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504135808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762011.838756:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762011.838758:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762011.838760:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504135808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762011.838764:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504135936:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762011.838774:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762011.838780:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762011/real 1789762011] req@ffff9018d070ca80 x1876699504135936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762011.838788:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504135936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762011.838793:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762011.838794:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762011.838796:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504135936:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762011.838800:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504136064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762011.838810:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762011.838816:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762011/real 1789762011] req@ffff9018d070ce00 x1876699504136064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762011.838823:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504136064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762011.838828:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762011.838830:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762011.838832:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504136064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762011.838835:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504136192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762011.838845:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762011.838850:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762011/real 1789762011] req@ffff9018d070d880 x1876699504136192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762011.838857:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504136192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762011.838862:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762011.838864:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762011.838866:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504136192:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762012.478188:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762013.502196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762014.526264:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762015.550275:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762016.574204:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762016.958151:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762016.958162:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762016.958165:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762016.958174:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1943 00000100:00080000:3.0:1789762016.958183:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762016.958189:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762016.958218:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504136320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762016.958245:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762016.958249:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762016.958250:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762016.958255:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1943 00000100:00080000:3.0:1789762016.958263:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762016.958269:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762016.958279:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762016.958294:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504136448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762016.958308:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762016.958313:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789762016.958314:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504136320:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762016.958315:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762016.958324:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1943 00000100:00080000:3.0:1789762016.958337:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762016.958341:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762016.958348:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762016.958364:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504136576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762016.958371:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762016.958374:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762016.958376:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789762016.958390:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762016.958448:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762016/real 1789762016] req@ffff9018d070d180 x1876699504136320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762016.958466:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1943 00000100:00100000:1.0:1789762016.958472:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504136320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762016.958473:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762016.958483:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789762016.958500:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762016.958506:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762016.958512:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504136320:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762016.958538:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504136448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762016.958566:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762016.958581:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762016/real 1789762016] req@ffff9018d070fb80 x1876699504136448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762016.958597:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504136448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762016.958607:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789762016.958611:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1789762016.958614:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762016.958622:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504136448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762016.958628:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504136704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762016.958636:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504136576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762016.958662:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762016.958677:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762016.958681:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762016.958683:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762016.958688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762016/real 1789762016] req@ffff9018d070e680 x1876699504136576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762016.958690:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1943 00000100:00080000:3.0:1789762016.958696:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762016.958704:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762016.958711:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762016.958718:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504136576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762016.958722:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504136832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762016.958753:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762016.958754:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762016.958755:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762016.958759:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762016.958764:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762016.958765:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1943 00000100:00080000:3.0:1789762016.958770:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762016.958775:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762016.958778:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762016.958783:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504136576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762016.958784:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504136960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762016.958817:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504136704:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762016.958843:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762016.958857:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762016/real 1789762016] req@ffff9018d070e300 x1876699504136704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762016.958870:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504136704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762016.958878:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762016.958882:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762016.958885:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504136704:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762016.958892:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504136832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762016.958914:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762016.958927:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762016/real 1789762016] req@ffff9018d070c000 x1876699504136832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762016.958939:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504136832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762016.958947:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762016.958951:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762016.958954:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504136832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762016.958961:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504136960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762016.958981:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762016.958993:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762016/real 1789762016] req@ffff9018d070dc00 x1876699504136960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762016.959006:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504136960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762016.959014:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762016.959018:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762016.959021:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504136960:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762017.598209:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762018.622217:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762019.646283:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762020.670175:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762021.694232:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762022.078175:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762022.078185:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762022.078189:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762022.078199:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1948 00000100:00080000:3.0:1789762022.078209:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762022.078217:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762022.078256:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504137088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762022.078287:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762022.078291:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762022.078293:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762022.078297:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1948 00000100:00080000:3.0:1789762022.078300:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762022.078303:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762022.078312:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762022.078323:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504137216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762022.078332:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762022.078335:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762022.078336:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762022.078343:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1948 00000100:00100000:1.0:1789762022.078345:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504137088:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762022.078346:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762022.078352:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762022.078358:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762022.078367:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504137344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762022.078374:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762022.078377:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762022.078378:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762022.078380:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1948 00000100:00080000:3.0:1789762022.078383:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762022.078399:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762022.078405:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762022.078419:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504137472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762022.078430:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762022.078435:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762022.078446:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762022.078449:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762022.078455:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1948 00000100:00080000:3.0:1789762022.078459:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762022.078463:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762022.078467:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762022.078485:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504137600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762022.078486:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762022/real 1789762022] req@ffff9018d070d500 x1876699504137088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762022.078502:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762022.078508:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762022.078510:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762022.078515:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1948 00000100:00080000:3.0:1789762022.078517:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762022.078522:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504137088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762022.078523:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762022.078528:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762022.078539:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504137728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762022.078542:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762022.078552:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762022.078558:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504137088:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762022.078584:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504137216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762022.078610:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762022.078623:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762022/real 1789762022] req@ffff9018d070f800 x1876699504137216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762022.078637:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504137216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762022.078652:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762022.078656:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762022.078660:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504137216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762022.078667:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504137344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762022.078687:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762022.078699:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762022/real 1789762022] req@ffff9018d070f100 x1876699504137344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762022.078712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504137344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762022.078720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762022.078724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762022.078728:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504137344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762022.078734:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504137472:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762022.078754:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762022.078766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762022/real 1789762022] req@ffff9018d070dc00 x1876699504137472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762022.078779:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504137472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762022.078787:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762022.078790:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762022.078794:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504137472:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762022.078799:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504137600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762022.078820:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762022.078831:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762022/real 1789762022] req@ffff9018d070c000 x1876699504137600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762022.078843:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504137600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762022.078851:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762022.078854:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762022.078857:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504137600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762022.078863:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504137728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762022.078883:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762022.078894:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762022/real 1789762022] req@ffff9018d070e300 x1876699504137728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762022.078906:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504137728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762022.078914:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762022.078917:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762022.078920:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504137728:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762022.718176:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762023.742246:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762024.766209:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762025.790244:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762026.814198:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762027.198138:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762027.198148:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762027.198152:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762027.198164:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1953 00000100:00080000:3.0:1789762027.198176:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762027.198183:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762027.198228:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504137856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762027.198260:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762027.198264:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762027.198265:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762027.198268:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1953 00000100:00080000:3.0:1789762027.198271:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762027.198276:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762027.198288:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762027.198304:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504137984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762027.198310:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504137856:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762027.198326:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762027.198329:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762027.198330:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762027.198335:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1953 00000100:00080000:3.0:1789762027.198338:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762027.198341:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762027.198346:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762027.198371:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504138112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762027.198384:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762027.198387:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762027.198389:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762027.198394:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1953 00000100:00080000:3.0:1789762027.198399:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762027.198403:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762027.198409:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762027.198424:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504138240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762027.198437:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762027.198441:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762027.198443:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762027.198447:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1953 00000100:00080000:3.0:1789762027.198451:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762027.198452:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504137984:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762027.198458:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762027.198464:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762027.198477:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504138368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762027.198486:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504138112:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762027.198486:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762027.198495:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762027.198497:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762027.198501:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1953 00000100:00080000:3.0:1789762027.198506:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762027.198508:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762027.198513:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762027.198514:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504138240:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762027.198524:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504138496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762027.198540:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504138368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762027.198559:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504138496:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762027.838186:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762028.862295:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762029.886221:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762030.270496:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789762030.270552:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762030.270575:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762030.270623:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762030.270636:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762030.270648:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762030.270666:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762030.270679:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789762030.270692:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762027/real 1789762030] req@ffff9018d070ce00 x1876699504138368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762030.270726:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504138368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762030.270740:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762030.270746:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762030.270752:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504138368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762030.270762:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762027/real 1789762030] req@ffff9018d070ca80 x1876699504138496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762030.270776:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504138496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762030.270785:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762030.270788:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762030.270792:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504138496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762030.270799:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762027/real 1789762030] req@ffff9018d070fb80 x1876699504137984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762030.270821:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504137984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762030.270830:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762030.270834:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762030.270838:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504137984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762030.270843:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762027/real 1789762030] req@ffff9018d070d180 x1876699504138112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762030.270855:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504138112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762030.270863:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762030.270867:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762030.270871:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504138112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762030.270878:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762027/real 1789762030] req@ffff9018d070d880 x1876699504138240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762030.270891:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504138240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762030.270899:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762030.270904:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762030.270908:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504138240:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762030.270915:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762027/real 1789762030] req@ffff9018d070e680 x1876699504137856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762030.270929:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504137856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762030.270938:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762030.270942:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762030.270945:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504137856:192.168.203.158@tcp:250:kworker.0 00010000:00010000:0.0:1789762030.910258:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762031.934247:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762032.318163:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762032.318171:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762032.318175:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762032.318184:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1958 00000100:00080000:2.0:1789762032.318195:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762032.318202:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762032.318249:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504138624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762032.318275:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762032.318278:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762032.318280:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762032.318283:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1958 00000100:00080000:2.0:1789762032.318286:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762032.318289:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762032.318296:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762032.318305:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504138752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762032.318335:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762032.318338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762032.318340:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762032.318343:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1958 00000100:00080000:2.0:1789762032.318347:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762032.318350:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762032.318357:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762032.318368:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504138880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762032.318378:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762032.318384:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762032.318385:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504138624:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762032.318393:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762032.318403:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1958 00000100:00080000:2.0:1789762032.318406:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762032.318408:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762032.318412:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762032.318422:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504139008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762032.318430:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762032.318432:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762032.318433:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762032.318436:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1958 00000100:00080000:2.0:1789762032.318438:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762032.318440:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762032.318444:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762032.318454:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504139136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762032.318465:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762032.318468:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762032.318470:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762032.318475:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1958 00000100:00080000:2.0:1789762032.318482:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762032.318486:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762032.318492:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762032.318500:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762032.318512:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504139264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762032.318568:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762032/real 1789762032] req@ffff9018d0f24380 x1876699504138624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762032.318591:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504138624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762032.318604:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762032.318609:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762032.318622:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504138624:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762032.318632:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504138752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762032.318663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762032.318680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762032/real 1789762032] req@ffff9018d0f27800 x1876699504138752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762032.318695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504138752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762032.318704:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762032.318708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762032.318712:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504138752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762032.318744:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504138880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762032.318768:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762032.318781:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762032/real 1789762032] req@ffff9018d0f25180 x1876699504138880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762032.318793:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504138880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762032.318800:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762032.318804:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762032.318807:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504138880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762032.318815:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504139008:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762032.318838:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762032.318851:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762032/real 1789762032] req@ffff9018d0f24a80 x1876699504139008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762032.318862:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504139008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762032.318871:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762032.318875:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762032.318878:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504139008:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762032.318885:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504139136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762032.318909:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762032.318922:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762032/real 1789762032] req@ffff9018d0f25500 x1876699504139136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762032.318934:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504139136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762032.318946:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762032.318950:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762032.318953:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504139136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762032.318959:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504139264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762032.318993:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762032.319012:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762032/real 1789762032] req@ffff9018d0f27480 x1876699504139264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762032.319027:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504139264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762032.319090:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762032.319100:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762032.319104:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504139264:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762032.958284:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762033.982288:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762035.006251:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762036.030215:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762037.054186:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762037.438188:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762037.438196:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762037.438200:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762037.438207:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1963 00000100:00080000:1.0:1789762037.438217:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762037.438223:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762037.438256:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504139392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762037.438292:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762037.438295:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762037.438296:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762037.438300:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1963 00000100:00080000:1.0:1789762037.438303:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762037.438306:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762037.438314:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762037.438324:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504139520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762037.438333:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762037.438335:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762037.438338:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762037.438340:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504139392:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762037.438344:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1963 00000100:00080000:1.0:1789762037.438349:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762037.438351:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762037.438355:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762037.438363:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504139648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762037.438370:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762037.438372:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762037.438374:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762037.438377:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1963 00000100:00080000:1.0:1789762037.438387:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762037.438390:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762037.438395:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789762037.438400:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762037.438404:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504139776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762037.438412:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762037.438416:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762037.438417:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762037.438420:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1963 00000100:00080000:1.0:1789762037.438422:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762037.438424:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762037.438427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762037.438437:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762037/real 1789762037] req@ffff9018c66a5880 x1876699504139392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762037.438437:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504139904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762037.438470:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762037.438473:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762037.438474:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762037.438477:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504139392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762037.438481:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1963 00000100:00080000:1.0:1789762037.438486:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762037.438492:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789762037.438495:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1789762037.438496:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789762037.438501:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762037.438509:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504139392:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762037.438509:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504140032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762037.438536:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504139520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762037.438552:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762037.438561:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762037/real 1789762037] req@ffff9018c66a6a00 x1876699504139520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762037.438569:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504139520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762037.438574:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762037.438576:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762037.438579:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504139520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762037.438582:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504139648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762037.438592:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762037.438597:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762037/real 1789762037] req@ffff9018c66a4a80 x1876699504139648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762037.438605:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504139648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762037.438609:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762037.438611:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762037.438613:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504139648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762037.438616:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504139776:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762037.438626:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762037.438631:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762037/real 1789762037] req@ffff9018c66a7480 x1876699504139776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762037.438638:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504139776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762037.438644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762037.438646:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762037.438648:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504139776:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762037.438651:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504139904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762037.438660:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762037.438666:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762037/real 1789762037] req@ffff9018c66a7100 x1876699504139904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762037.438672:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504139904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762037.438677:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762037.438678:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762037.438680:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504139904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762037.438683:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504140032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762037.438693:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762037.438698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762037/real 1789762037] req@ffff9018c66a4380 x1876699504140032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762037.438704:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504140032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762037.438709:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762037.438710:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762037.438713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504140032:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762038.078259:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762039.102297:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762040.126171:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762041.150294:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762042.174375:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762042.558223:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762042.558244:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762042.558248:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762042.558259:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1969 00000100:00080000:2.0:1789762042.558271:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762042.558279:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762042.558328:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504140160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762042.558353:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762042.558356:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762042.558357:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762042.558360:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1969 00000100:00080000:2.0:1789762042.558362:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762042.558365:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762042.558375:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762042.558385:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504140288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762042.558393:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762042.558395:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762042.558397:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762042.558403:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504140160:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762042.558404:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1969 00000100:00080000:2.0:1789762042.558409:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762042.558413:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762042.558417:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762042.558428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504140416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762042.558434:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762042.558437:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762042.558438:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762042.558441:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1969 00000100:00080000:2.0:1789762042.558443:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762042.558446:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762042.558450:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762042.558458:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504140544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762042.558464:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762042.558467:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762042.558468:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762042.558470:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1969 00000100:00080000:2.0:1789762042.558473:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762042.558475:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762042.558479:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762042.558482:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762042.558491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504140672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762042.558499:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762042.558501:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762042.558502:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762042.558505:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1969 00000100:00080000:2.0:1789762042.558509:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762042.558512:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762042.558515:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762042.558523:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504140800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762042.558533:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762042/real 1789762042] req@ffff9018d0f26680 x1876699504140160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762042.558550:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504140160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762042.558559:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762042.558562:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762042.558565:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504140160:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762042.558588:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504140288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762042.558604:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762042.558613:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762042/real 1789762042] req@ffff9018d0f24700 x1876699504140288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762042.558621:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504140288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762042.558627:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762042.558629:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762042.558631:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504140288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762042.558634:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504140416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762042.558647:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762042.558652:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762042/real 1789762042] req@ffff9018d0f24000 x1876699504140416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762042.558660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504140416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762042.558665:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762042.558667:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762042.558669:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504140416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762042.558672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504140544:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762042.558682:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762042.558688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762042/real 1789762042] req@ffff9018d0f24e00 x1876699504140544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762042.558695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504140544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762042.558701:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762042.558703:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762042.558704:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504140544:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762042.558708:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504140672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762042.558717:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762042.558723:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762042/real 1789762042] req@ffff9018d0f27480 x1876699504140672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762042.558734:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504140672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762042.558739:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762042.558741:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762042.558743:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504140672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762042.558746:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504140800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762042.558755:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762042.558761:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762042/real 1789762042] req@ffff9018d0f25500 x1876699504140800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762042.558767:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504140800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762042.558772:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762042.558774:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762042.558776:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504140800:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762043.198264:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762044.222193:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762045.246237:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762046.270323:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762047.294189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762047.678189:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762047.678220:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762047.678225:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762047.678238:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1974 00000100:00080000:1.0:1789762047.678253:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762047.678263:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762047.678314:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504140928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762047.678345:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762047.678349:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762047.678351:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762047.678356:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1974 00000100:00080000:1.0:1789762047.678360:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762047.678365:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762047.678377:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762047.678394:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504141056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762047.678397:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504140928:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762047.678425:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762047.678429:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762047.678431:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762047.678437:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1974 00000100:00080000:1.0:1789762047.678442:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762047.678447:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762047.678452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762047.678466:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504141184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762047.678475:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762047.678478:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762047.678483:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762047.678486:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762047.678491:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1974 00000100:00080000:1.0:1789762047.678496:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762047.678500:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762047.678507:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762047.678520:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504141312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762047.678532:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762047/real 1789762047] req@ffff9018c66a5180 x1876699504140928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762047.678533:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762047.678543:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762047.678547:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762047.678556:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1974 00000100:00080000:1.0:1789762047.678561:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762047.678570:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762047.678571:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504140928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762047.678577:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789762047.678591:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789762047.678597:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504141440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762047.678601:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762047.678610:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504140928:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762047.678620:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762047.678637:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762047.678639:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762047.678644:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1974 00000100:00080000:1.0:1789762047.678649:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762047.678652:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504141056:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789762047.678658:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762047.678665:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762047.678680:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762047.678684:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504141568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762047.678695:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762047/real 1789762047] req@ffff9018c66a6680 x1876699504141056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762047.678707:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504141056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762047.678721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762047.678724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762047.678726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504141056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762047.678732:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504141184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762047.678746:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762047.678753:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762047/real 1789762047] req@ffff9018c66a4700 x1876699504141184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762047.678761:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504141184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762047.678766:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762047.678769:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762047.678771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504141184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762047.678775:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504141312:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762047.678786:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762047.678792:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762047/real 1789762047] req@ffff9018c66a6300 x1876699504141312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762047.678800:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504141312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762047.678805:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762047.678807:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762047.678809:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504141312:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762047.678812:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504141440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762047.678822:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762047.678828:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762047/real 1789762047] req@ffff9018c66a4e00 x1876699504141440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762047.678834:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504141440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762047.678840:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762047.678842:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762047.678844:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504141440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762047.678856:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504141568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762047.678867:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762047.678873:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762047/real 1789762047] req@ffff9018c66a5500 x1876699504141568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762047.678881:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504141568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762047.678886:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762047.678888:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762047.678890:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504141568:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762048.318261:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762049.342205:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762050.366216:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762051.390283:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762052.414237:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762052.798177:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762052.798190:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762052.798196:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762052.798210:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1979 00000100:00080000:1.0:1789762052.798225:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762052.798232:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762052.798271:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504141696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762052.798300:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762052.798304:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762052.798306:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762052.798309:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1979 00000100:00080000:1.0:1789762052.798313:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762052.798316:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762052.798327:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762052.798343:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504141824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762052.798347:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504141696:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762052.798362:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762052.798366:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762052.798367:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762052.798372:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1979 00000100:00080000:1.0:1789762052.798374:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762052.798378:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762052.798381:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762052.798393:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504141952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762052.798400:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762052.798409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762052.798412:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762052.798415:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1979 00000100:00080000:1.0:1789762052.798421:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762052.798424:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762052.798428:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762052.798433:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762052.798445:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504142080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762052.798453:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762052.798456:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762052.798457:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762052.798461:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1979 00000100:00080000:1.0:1789762052.798466:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762052.798471:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762052.798471:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762052/real 1789762052] req@ffff9018c66a7800 x1876699504141696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762052.798483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762052.798506:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504142208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762052.798508:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504141696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762052.798550:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762052.798568:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762052.798571:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762052.798573:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762052.798578:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1979 00000100:00080000:2.0:1789762052.798583:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789762052.798584:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762052.798593:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762052.798598:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504141696:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789762052.798600:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762052.798622:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504142336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762052.798638:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504141824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762052.798660:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762052.798673:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762052/real 1789762052] req@ffff9018c66a7b80 x1876699504141824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762052.798688:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504141824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762052.798698:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762052.798703:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762052.798707:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504141824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762052.798716:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504141952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762052.798739:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762052.798752:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762052/real 1789762052] req@ffff9018c66a4000 x1876699504141952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762052.798762:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504141952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762052.798773:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762052.798779:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762052.798786:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504141952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762052.798796:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504142080:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762052.798826:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762052.798847:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762052/real 1789762052] req@ffff9018c66a5f80 x1876699504142080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762052.798859:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504142080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762052.798866:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762052.798869:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762052.798872:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504142080:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762052.798878:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504142208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762052.798898:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762052.798906:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762052/real 1789762052] req@ffff9018c66a6d80 x1876699504142208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762052.798914:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504142208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762052.798919:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762052.798921:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762052.798923:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504142208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762052.798927:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504142336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762052.798937:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762052.798948:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762052/real 1789762052] req@ffff9018c66a5c00 x1876699504142336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762052.798956:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504142336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762052.798966:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762052.798969:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762052.798971:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504142336:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762053.438248:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762054.462231:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762055.486154:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762056.510206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762057.534255:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762057.918212:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762057.918224:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762057.918228:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762057.918239:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1984 00000100:00080000:3.0:1789762057.918251:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762057.918259:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762057.918301:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504142464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762057.918326:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762057.918329:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762057.918331:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762057.918334:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1984 00000100:00080000:3.0:1789762057.918336:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762057.918339:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762057.918348:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762057.918357:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504142592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762057.918365:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762057.918367:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762057.918369:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762057.918372:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1984 00000100:00080000:3.0:1789762057.918374:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762057.918376:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762057.918379:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762057.918385:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504142720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762057.918403:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762057.918405:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504142464:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762057.918408:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762057.918410:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762057.918418:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1984 00000100:00080000:3.0:1789762057.918428:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762057.918435:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762057.918440:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762057.918457:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504142848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762057.918467:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762057.918472:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762057.918473:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762057.918477:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1984 00000100:00080000:3.0:1789762057.918486:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762057.918491:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789762057.918498:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789762057.918503:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762057.918521:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504142976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762057.918530:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762057.918533:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762057.918535:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762057.918543:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1984 00000100:00100000:2.0:1789762057.918544:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762057/real 1789762057] req@ffff9018d070df80 x1876699504142464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762057.918558:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762057.918565:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762057.918570:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504142464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789762057.918574:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762057.918589:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762057.918595:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762057.918602:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504143104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762057.918603:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504142464:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762057.918624:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504142592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762057.918648:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762057.918666:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762057/real 1789762057] req@ffff9018d070ed80 x1876699504142592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762057.918675:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504142592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762057.918681:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762057.918684:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762057.918686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504142592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762057.918709:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504142720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762057.918721:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762057.918727:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762057/real 1789762057] req@ffff9018d070f480 x1876699504142720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762057.918735:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504142720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762057.918740:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762057.918742:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762057.918744:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504142720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762057.918748:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504142848:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762057.918759:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762057.918774:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762057/real 1789762057] req@ffff9018d070e680 x1876699504142848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762057.918785:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504142848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762057.918791:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762057.918794:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762057.918796:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504142848:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762057.918800:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504142976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762057.918820:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762057.918831:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762057/real 1789762057] req@ffff9018d070d880 x1876699504142976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762057.918845:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504142976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762057.918854:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762057.918858:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762057.918861:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504142976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762057.918868:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504143104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762057.918884:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762057.918891:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762057/real 1789762057] req@ffff9018d070d180 x1876699504143104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762057.918899:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504143104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762057.918904:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762057.918907:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762057.918909:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504143104:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762058.558247:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762059.582253:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762060.606180:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762061.630146:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762062.654210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762063.038161:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762063.038174:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762063.038178:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762063.038188:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1989 00000100:00080000:3.0:1789762063.038200:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762063.038208:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762063.038249:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504143232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762063.038276:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762063.038280:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762063.038282:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762063.038286:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1989 00000100:00080000:3.0:1789762063.038290:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762063.038294:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762063.038306:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762063.038319:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504143360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762063.038328:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504143232:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762063.038330:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762063.038335:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762063.038336:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762063.038339:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1989 00000100:00080000:3.0:1789762063.038343:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762063.038347:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762063.038351:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762063.038359:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504143488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762063.038368:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762063.038371:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762063.038373:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762063.038377:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1989 00000100:00080000:3.0:1789762063.038390:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762063.038394:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762063.038399:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789762063.038403:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762063.038409:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504143616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762063.038419:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762063.038422:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762063.038423:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762063.038427:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1989 00000100:00080000:3.0:1789762063.038430:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762063.038434:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762063.038438:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762063.038447:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504143744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762063.038451:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762063/real 1789762063] req@ffff9018d070fb80 x1876699504143232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762063.038457:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762063.038460:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762063.038462:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762063.038465:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1989 00000100:00080000:3.0:1789762063.038468:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762063.038470:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504143232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762063.038471:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762063.038476:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762063.038482:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789762063.038484:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504143872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762063.038487:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762063.038492:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504143232:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762063.038519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504143360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762063.038544:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762063.038558:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762063/real 1789762063] req@ffff9018d070ca80 x1876699504143360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762063.038568:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504143360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762063.038573:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762063.038575:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762063.038578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504143360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762063.038581:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504143488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762063.038596:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762063.038603:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762063/real 1789762063] req@ffff9018d070ce00 x1876699504143488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762063.038611:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504143488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762063.038620:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762063.038622:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762063.038624:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504143488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762063.038627:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504143616:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762063.038643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762063.038654:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762063/real 1789762063] req@ffff9018d070e300 x1876699504143616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762063.038665:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504143616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762063.038674:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762063.038678:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762063.038681:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504143616:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762063.038687:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504143744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762063.038704:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762063.038712:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762063/real 1789762063] req@ffff9018d070c000 x1876699504143744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762063.038720:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504143744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762063.038725:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762063.038727:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762063.038729:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504143744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762063.038733:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504143872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762063.038744:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762063.038749:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762063/real 1789762063] req@ffff9018d070dc00 x1876699504143872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762063.038756:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504143872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762063.038763:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762063.038766:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762063.038769:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504143872:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762063.678163:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762064.702204:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762065.726203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762066.750238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762067.774268:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762068.158107:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762068.158118:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762068.158122:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762068.158133:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1994 00000100:00080000:1.0:1789762068.158144:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762068.158152:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762068.158184:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504144000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762068.158206:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762068.158209:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762068.158211:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762068.158214:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1994 00000100:00080000:1.0:1789762068.158217:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762068.158225:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762068.158233:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762068.158243:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504144128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762068.158250:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762068.158253:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762068.158256:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762068.158258:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1994 00000100:00080000:1.0:1789762068.158260:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762068.158262:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762068.158265:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762068.158271:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504144256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762068.158277:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762068.158280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762068.158282:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762068.158285:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1994 00000100:00080000:1.0:1789762068.158287:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762068.158289:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762068.158293:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762068.158307:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504144384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762068.158308:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504144000:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762068.158336:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762068.158340:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762068.158342:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762068.158346:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1994 00000100:00080000:1.0:1789762068.158350:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762068.158355:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762068.158361:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762068.158375:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504144512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762068.158387:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762068.158392:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762068.158393:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762068.158397:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1994 00000100:00080000:1.0:1789762068.158401:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762068.158404:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762068.158409:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762068.158418:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504144640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762068.158435:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762068.158490:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762068/real 1789762068] req@ffff9018c66a5c00 x1876699504144000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762068.158514:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504144000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762068.158526:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762068.158530:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762068.158540:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504144000:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762068.158550:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504144128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762068.158582:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762068.158598:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762068/real 1789762068] req@ffff9018c66a6d80 x1876699504144128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762068.158612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504144128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762068.158621:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762068.158625:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762068.158629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504144128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762068.158636:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504144256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762068.158658:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762068.158672:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762068/real 1789762068] req@ffff9018c66a5f80 x1876699504144256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762068.158685:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504144256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762068.158693:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762068.158697:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762068.158701:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504144256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762068.158736:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504144384:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762068.158759:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762068.158772:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762068/real 1789762068] req@ffff9018c66a4000 x1876699504144384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762068.158791:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504144384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762068.158799:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762068.158801:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762068.158805:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504144384:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762068.158811:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504144512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762068.158831:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762068.158843:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762068/real 1789762068] req@ffff9018c66a7b80 x1876699504144512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762068.158855:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504144512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762068.158863:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762068.158867:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762068.158871:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504144512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762068.158877:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504144640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762068.158898:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762068.158911:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762068/real 1789762068] req@ffff9018c66a7800 x1876699504144640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762068.158924:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504144640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762068.158940:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762068.158943:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762068.158947:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504144640:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762068.798169:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762069.822211:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762070.846169:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762071.870226:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762072.894269:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762073.278150:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762073.278162:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762073.278167:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762073.278178:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 1999 00000100:00080000:1.0:1789762073.278190:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762073.278196:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762073.278241:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504144768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762073.278266:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762073.278269:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762073.278270:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762073.278274:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1999 00000100:00080000:1.0:1789762073.278276:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762073.278278:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762073.278287:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762073.278297:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504144896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762073.278304:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762073.278307:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762073.278308:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762073.278311:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1999 00000100:00080000:1.0:1789762073.278313:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762073.278315:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762073.278318:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762073.278326:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504144768:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762073.278327:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504145024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762073.278347:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762073.278349:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762073.278351:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762073.278353:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 1999 00000100:00080000:1.0:1789762073.278355:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762073.278357:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762073.278361:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762073.278368:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504145152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762073.278375:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762073.278378:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762073.278379:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762073.278381:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1999 00000100:00080000:1.0:1789762073.278384:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762073.278387:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762073.278389:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762073.278397:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504145280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762073.278404:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762073.278408:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762073.278410:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762073.278416:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762073.278418:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 1999 00000100:00080000:1.0:1789762073.278425:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762073.278430:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762073.278436:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762073.278448:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504145408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762073.278477:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762073/real 1789762073] req@ffff9018c66a5500 x1876699504144768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762073.278499:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504144768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762073.278520:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762073.278524:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762073.278529:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504144768:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762073.278537:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504144896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762073.278560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762073.278573:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762073/real 1789762073] req@ffff9018c66a4e00 x1876699504144896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762073.278585:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504144896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762073.278595:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762073.278599:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762073.278602:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504144896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762073.278630:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504145024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762073.278648:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762073.278658:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762073/real 1789762073] req@ffff9018c66a6300 x1876699504145024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762073.278670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504145024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762073.278679:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762073.278682:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762073.278685:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504145024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762073.278691:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504145152:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762073.278708:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762073.278717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762073/real 1789762073] req@ffff9018c66a4700 x1876699504145152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762073.278729:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504145152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762073.278738:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762073.278741:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762073.278744:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504145152:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762073.278750:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504145280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762073.278767:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762073.278776:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762073/real 1789762073] req@ffff9018c66a6680 x1876699504145280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762073.278789:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504145280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762073.278797:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762073.278800:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762073.278803:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504145280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762073.278809:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504145408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762073.278826:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762073.278834:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762073/real 1789762073] req@ffff9018c66a5180 x1876699504145408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762073.278846:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504145408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762073.278854:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762073.278857:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762073.278861:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504145408:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762073.918234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762074.942203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762075.966226:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762076.990241:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762078.014252:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762078.398205:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762078.398217:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762078.398221:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762078.398251:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2004 00000100:00080000:1.0:1789762078.398266:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762078.398277:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762078.398348:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504145536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762078.398396:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762078.398399:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762078.398401:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762078.398404:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2004 00000100:00080000:1.0:1789762078.398407:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762078.398411:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762078.398421:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762078.398432:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504145664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762078.398439:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762078.398442:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762078.398444:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762078.398447:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2004 00000100:00080000:1.0:1789762078.398450:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762078.398452:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762078.398455:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762078.398468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504145792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762078.398470:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504145536:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762078.398483:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762078.398485:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762078.398486:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762078.398489:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2004 00000100:00080000:1.0:1789762078.398492:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762078.398494:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762078.398498:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762078.398508:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504145920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762078.398515:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762078.398518:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762078.398519:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762078.398522:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2004 00000100:00080000:1.0:1789762078.398524:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762078.398527:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762078.398531:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762078.398537:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504146048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762078.398544:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762078.398546:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762078.398547:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762078.398550:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2004 00000100:00080000:1.0:1789762078.398552:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762078.398554:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762078.398557:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762078.398562:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762078.398567:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504146176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762078.398618:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762078/real 1789762078] req@ffff9018c66a4380 x1876699504145536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762078.398641:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504145536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762078.398649:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762078.398653:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762078.398656:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504145536:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762078.398662:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504145664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762078.398681:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762078.398690:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762078/real 1789762078] req@ffff9018c66a7100 x1876699504145664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762078.398698:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504145664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762078.398704:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762078.398706:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762078.398708:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504145664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762078.398732:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504145792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762078.398743:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762078.398749:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762078/real 1789762078] req@ffff9018c66a7480 x1876699504145792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762078.398756:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504145792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762078.398766:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762078.398768:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762078.398769:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504145792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762078.398773:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504145920:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762078.398783:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762078.398789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762078/real 1789762078] req@ffff9018c66a4a80 x1876699504145920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762078.398795:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504145920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762078.398801:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762078.398803:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762078.398804:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504145920:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762078.398808:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504146048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762078.398818:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762078.398824:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762078/real 1789762078] req@ffff9018c66a6a00 x1876699504146048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762078.398830:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504146048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762078.398835:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762078.398837:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762078.398839:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504146048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762078.398843:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504146176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762078.398852:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762078.398857:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762078/real 1789762078] req@ffff9018c66a5880 x1876699504146176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762078.398864:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504146176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762078.398869:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762078.398871:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762078.398872:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504146176:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762079.038226:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762080.062203:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762081.086190:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762082.110220:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762083.134147:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762083.518127:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762083.518138:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762083.518142:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762083.518154:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2010 00000100:00080000:1.0:1789762083.518165:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762083.518172:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762083.518217:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504146304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762083.518244:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762083.518247:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762083.518248:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762083.518257:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2010 00000100:00080000:1.0:1789762083.518260:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762083.518263:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762083.518273:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762083.518285:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504146432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762083.518312:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762083.518316:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762083.518318:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762083.518325:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2010 00000100:00080000:1.0:1789762083.518329:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762083.518338:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504146304:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762083.518340:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762083.518354:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762083.518380:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504146560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762083.518407:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762083.518412:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762083.518415:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762083.518420:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2010 00000100:00080000:1.0:1789762083.518428:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762083.518434:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789762083.518441:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:1.0:1789762083.518442:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762083.518466:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504146688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762083.518485:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762083.518488:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762083.518490:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762083.518496:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2010 00000100:00100000:2.0:1789762083.518498:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762083/real 1789762083] req@ffff9018c66a5880 x1876699504146304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762083.518501:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762083.518512:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762083.518534:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762083.518556:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504146816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762083.518565:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762083.518568:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762083.518571:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762083.518583:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2010 00000100:00100000:2.0:1789762083.518585:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504146304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762083.518588:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762083.518599:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762083.518623:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762083.518640:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789762083.518643:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504146944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762083.518651:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762083.518667:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504146304:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762083.518678:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504146432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762083.518728:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762083.518743:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762083/real 1789762083] req@ffff9018c66a6a00 x1876699504146432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762083.518755:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504146432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762083.518764:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762083.518768:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762083.518771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504146432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762083.518797:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504146560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762083.518814:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762083.518823:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762083/real 1789762083] req@ffff9018c66a4a80 x1876699504146560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762083.518867:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504146560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762083.518881:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762083.518886:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762083.518892:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504146560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762083.518918:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504146688:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762083.518962:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762083.518992:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762083/real 1789762083] req@ffff9018c66a7480 x1876699504146688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762083.519006:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504146688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762083.519012:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762083.519015:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762083.519017:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504146688:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762083.519022:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504146816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762083.519035:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762083.519041:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762083/real 1789762083] req@ffff9018c66a7100 x1876699504146816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762083.519108:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504146816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762083.519117:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762083.519120:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762083.519124:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504146816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762083.519129:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504146944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762083.519151:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762083.519161:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762083/real 1789762083] req@ffff9018c66a4380 x1876699504146944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762083.519171:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504146944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762083.519202:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762083.519205:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762083.519208:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504146944:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762084.158175:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762085.182241:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762086.206213:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762087.230207:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762088.254194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762088.638293:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762088.638305:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762088.638312:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762088.638327:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2015 00000100:00080000:2.0:1789762088.638344:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762088.638354:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762088.638407:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504147072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762088.638447:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762088.638452:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762088.638455:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762088.638461:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2015 00000100:00080000:2.0:1789762088.638468:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762088.638473:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762088.638486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762088.638503:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504147200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762088.638518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762088.638527:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789762088.638531:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504147072:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762088.638531:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762088.638541:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2015 00000100:00080000:2.0:1789762088.638551:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762088.638556:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762088.638563:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762088.638577:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504147328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762088.638589:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762088.638594:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762088.638597:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762088.638601:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2015 00000100:00080000:2.0:1789762088.638605:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762088.638609:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762088.638614:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762088.638626:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504147456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762088.638638:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762088.638651:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762088.638657:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762088.638659:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762088.638665:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2015 00000100:00080000:2.0:1789762088.638673:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762088.638678:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762088.638686:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762088.638698:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504147584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762088.638710:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762088/real 1789762088] req@ffff9018d0f24a80 x1876699504147072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762088.638715:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762088.638729:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762088.638736:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762088.638750:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2015 00000100:00080000:2.0:1789762088.638759:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762088.638767:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762088.638775:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762088.638795:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504147712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762088.638809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504147072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762088.638826:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762088.638833:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762088.638839:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504147072:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762088.638873:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504147200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762088.638924:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762088.638955:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762088/real 1789762088] req@ffff9018d0f25180 x1876699504147200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762088.638973:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504147200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762088.638983:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762088.638988:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762088.638994:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504147200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762088.639005:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504147328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762088.639099:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762088.639134:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762088/real 1789762088] req@ffff9018d0f27800 x1876699504147328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762088.639181:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504147328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762088.639207:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762088.639211:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762088.639214:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504147328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762088.639222:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504147456:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762088.639252:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762088.639277:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762088/real 1789762088] req@ffff9018d0f24380 x1876699504147456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762088.639289:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504147456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762088.639296:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762088.639298:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762088.639304:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504147456:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762088.639314:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504147584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762088.639355:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762088.639371:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762088/real 1789762088] req@ffff9018d0f26d80 x1876699504147584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762088.639384:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504147584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762088.639393:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762088.639396:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762088.639400:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504147584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762088.639408:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504147712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762088.639430:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762088.639439:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762088/real 1789762088] req@ffff9018d0f25f80 x1876699504147712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762088.639448:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504147712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762088.639453:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762088.639455:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762088.639458:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504147712:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762089.278226:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762090.302196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762091.326296:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762092.350226:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762093.374182:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762093.758096:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762093.758105:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762093.758108:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762093.758116:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2020 00000100:00080000:2.0:1789762093.758127:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762093.758135:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762093.758174:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504147840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762093.758199:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762093.758202:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762093.758204:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762093.758208:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2020 00000100:00080000:2.0:1789762093.758211:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762093.758214:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762093.758223:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762093.758233:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504147968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762093.758241:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762093.758243:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762093.758244:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762093.758247:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2020 00000100:00080000:2.0:1789762093.758249:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762093.758260:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762093.758262:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762093.758269:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504147840:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762093.758270:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504148096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762093.758283:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762093.758285:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762093.758287:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762093.758289:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2020 00000100:00080000:2.0:1789762093.758292:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762093.758294:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762093.758297:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762093.758306:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504148224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762093.758314:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762093.758317:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762093.758318:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762093.758320:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2020 00000100:00080000:2.0:1789762093.758325:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762093.758327:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762093.758330:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762093.758338:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504148352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762093.758345:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762093.758358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762093.758359:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762093.758361:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2020 00000100:00080000:2.0:1789762093.758363:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762093.758366:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762093.758369:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762093.758375:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504147968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762093.758379:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504148480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762093.758411:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504148096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762093.758427:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504148224:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762093.758468:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504148352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762093.758494:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504148480:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762094.398213:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762095.422326:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762096.446226:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762096.830283:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:3.0:1789762096.830317:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762096.830341:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762096.830374:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762096.830380:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762096.830384:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762096.830390:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762096.830395:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789762096.830451:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762093/real 1789762096] req@ffff9018d0f26300 x1876699504148096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762096.830471:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504148096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762096.830481:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762096.830485:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762096.830489:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504148096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762096.830494:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762093/real 1789762096] req@ffff9018d0f27100 x1876699504148224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762096.830503:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504148224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762096.830508:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762096.830510:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762096.830512:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504148224:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762096.830516:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762093/real 1789762096] req@ffff9018d0f25f80 x1876699504148352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762096.830523:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504148352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762096.830527:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762096.830529:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762096.830531:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504148352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762096.830535:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762093/real 1789762096] req@ffff9018d0f26d80 x1876699504148480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762096.830541:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504148480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762096.830547:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762096.830550:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762096.830552:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504148480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762096.830555:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762093/real 1789762096] req@ffff9018d0f25c00 x1876699504147840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762096.830562:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504147840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762096.830566:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762096.830568:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762096.830570:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504147840:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762096.830574:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762093/real 1789762096] req@ffff9018d0f26a00 x1876699504147968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762096.830580:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504147968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762096.830584:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762096.830587:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762096.830588:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504147968:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762097.470210:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762098.494260:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762098.878212:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762098.878222:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762098.878226:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762098.878236:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2025 00000100:00080000:0.0:1789762098.878249:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762098.878257:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762098.878298:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504148608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762098.878337:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762098.878342:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762098.878344:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762098.878349:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2025 00000100:00080000:0.0:1789762098.878352:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762098.878357:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762098.878369:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762098.878386:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504148736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762098.878400:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504148608:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762098.878402:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762098.878411:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762098.878414:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762098.878419:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2025 00000100:00080000:0.0:1789762098.878424:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762098.878428:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762098.878434:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762098.878448:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504148864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762098.878458:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762098.878462:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762098.878465:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762098.878469:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2025 00000100:00080000:0.0:1789762098.878483:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762098.878500:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762098.878507:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762098.878515:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762098.878530:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504148992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762098.878541:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762098.878545:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762098.878547:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762098.878551:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2025 00000100:00080000:0.0:1789762098.878556:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762098.878558:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762098/real 1789762098] req@ffff9018c8b9a300 x1876699504148608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762098.878564:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762098.878575:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762098.878590:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504149120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762098.878600:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504148608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762098.878604:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762098.878614:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762098.878618:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762098.878625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762098.878626:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2025 00000100:00080000:0.0:1789762098.878635:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762098.878637:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789762098.878645:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762098.878648:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504148608:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789762098.878654:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762098.878674:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504149248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762098.878690:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504148736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762098.878722:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762098.878738:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762098/real 1789762098] req@ffff9018c8b9ad80 x1876699504148736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762098.878750:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504148736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762098.878757:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762098.878760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762098.878764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504148736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762098.878770:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504148864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762098.878790:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762098.878802:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762098/real 1789762098] req@ffff9018c8b99500 x1876699504148864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762098.878830:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504148864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762098.878845:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762098.878849:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762098.878852:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504148864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762098.878859:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504148992:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762098.878883:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762098.878896:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762098/real 1789762098] req@ffff9018c8b99c00 x1876699504148992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762098.878909:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504148992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762098.878918:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762098.878922:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762098.878926:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504148992:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762098.878932:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504149120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762098.878955:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762098.878966:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762098/real 1789762098] req@ffff9018c8b98380 x1876699504149120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762098.878978:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504149120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762098.878987:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762098.878990:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762098.878995:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504149120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762098.879021:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504149248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762098.879083:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762098.879097:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762098/real 1789762098] req@ffff9018c8b9a680 x1876699504149248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762098.879110:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504149248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762098.879118:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762098.879122:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762098.879126:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504149248:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762099.518203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762100.542313:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762101.566309:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762102.590250:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762103.614222:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762103.998128:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762103.998139:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762103.998144:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762103.998154:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2030 00000100:00080000:0.0:1789762103.998167:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762103.998174:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762103.998217:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504149376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762103.998244:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762103.998247:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762103.998249:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762103.998252:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2030 00000100:00080000:0.0:1789762103.998261:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762103.998264:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762103.998274:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762103.998283:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504149504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762103.998291:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762103.998294:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762103.998295:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762103.998298:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2030 00000100:00080000:0.0:1789762103.998302:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762103.998323:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762103.998327:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762103.998334:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504149632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762103.998341:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762103.998346:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762103.998348:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762103.998348:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504149376:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762103.998355:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2030 00000100:00080000:0.0:1789762103.998360:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762103.998362:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762103.998365:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762103.998373:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504149760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762103.998380:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762103.998383:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762103.998384:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762103.998387:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2030 00000100:00080000:0.0:1789762103.998389:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762103.998392:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762103.998394:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762103.998401:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504149888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762103.998412:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762103.998415:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762103.998416:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762103.998418:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2030 00000100:00080000:0.0:1789762103.998423:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762103.998425:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762103.998428:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762103.998435:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504150016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762103.998443:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762103.998509:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762103/real 1789762103] req@ffff9018c8b9b800 x1876699504149376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762103.998525:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504149376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762103.998533:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762103.998536:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762103.998540:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504149376:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762103.998546:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504149504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762103.998580:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762103.998590:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762103/real 1789762103] req@ffff9018c8b9aa00 x1876699504149504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762103.998598:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504149504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762103.998604:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762103.998606:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762103.998609:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504149504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762103.998631:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504149632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762103.998643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762103.998649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762103/real 1789762103] req@ffff9018c8b98000 x1876699504149632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762103.998656:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504149632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762103.998662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762103.998663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762103.998665:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504149632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762103.998669:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504149760:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762103.998679:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762103.998685:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762103/real 1789762103] req@ffff9018c8b9b480 x1876699504149760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762103.998693:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504149760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762103.998699:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762103.998701:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762103.998703:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504149760:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762103.998706:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504149888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762103.998717:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762103.998722:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762103/real 1789762103] req@ffff9018c8b9b100 x1876699504149888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762103.998730:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504149888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762103.998746:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762103.998747:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762103.998749:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504149888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762103.998753:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504150016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762103.998763:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762103.998768:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762103/real 1789762103] req@ffff9018c8b98e00 x1876699504150016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762103.998792:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504150016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762103.998797:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762103.998800:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762103.998802:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504150016:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762104.638209:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762105.662192:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762106.686230:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762107.710169:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762108.734250:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762109.118133:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762109.118141:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762109.118145:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762109.118153:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2035 00000100:00080000:0.0:1789762109.118161:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762109.118167:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762109.118201:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504150144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762109.118235:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762109.118238:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762109.118240:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762109.118242:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2035 00000100:00080000:0.0:1789762109.118245:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762109.118248:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762109.118256:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762109.118267:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504150272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762109.118275:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762109.118288:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762109.118289:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762109.118292:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2035 00000100:00080000:0.0:1789762109.118295:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762109.118297:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762109.118301:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762109.118309:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504150144:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762109.118316:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504150400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762109.118333:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762109.118336:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762109.118338:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762109.118343:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2035 00000100:00080000:0.0:1789762109.118347:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762109.118352:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762109.118358:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762109.118374:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504150528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762109.118389:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762109.118392:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762109.118393:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762109.118396:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2035 00000100:00080000:0.0:1789762109.118400:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762109.118404:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762109.118419:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762109.118426:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504150656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762109.118432:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762109.118434:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762109.118435:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762109.118437:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2035 00000100:00080000:0.0:1789762109.118440:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762109.118446:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789762109.118449:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789762109.118452:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762109.118461:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504150784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762109.118529:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762109/real 1789762109] req@ffff9018c8b98a80 x1876699504150144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762109.118566:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504150144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762109.118580:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762109.118585:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762109.118590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504150144:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762109.118639:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504150272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762109.118667:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762109.118679:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762109/real 1789762109] req@ffff9018c8b99880 x1876699504150272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762109.118694:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504150272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762109.118705:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762109.118709:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762109.118714:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504150272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762109.118722:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504150400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762109.118790:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762109.118810:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762109/real 1789762109] req@ffff9018c8b9bb80 x1876699504150400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762109.118823:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504150400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762109.118832:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762109.118836:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762109.118839:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504150400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762109.118847:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504150528:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762109.118866:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762109.118874:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762109/real 1789762109] req@ffff9018c8b98700 x1876699504150528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762109.118882:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504150528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762109.118887:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762109.118889:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762109.118891:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504150528:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762109.118895:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504150656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762109.118906:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762109.118912:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762109/real 1789762109] req@ffff9018c8b99f80 x1876699504150656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762109.118919:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504150656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762109.118924:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762109.118927:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762109.118929:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504150656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762109.118949:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504150784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762109.118963:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762109.118970:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762109/real 1789762109] req@ffff9018c8b98e00 x1876699504150784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762109.118978:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504150784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762109.118983:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762109.118986:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762109.118988:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504150784:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762109.758219:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762110.782366:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762111.806202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762112.830228:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762113.854272:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762114.238149:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762114.238159:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762114.238163:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762114.238172:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2040 00000100:00080000:0.0:1789762114.238183:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762114.238190:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762114.238226:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504150912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762114.238248:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762114.238251:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762114.238253:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762114.238256:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2040 00000100:00080000:0.0:1789762114.238259:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762114.238262:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762114.238277:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762114.238287:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504151040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762114.238298:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762114.238301:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789762114.238301:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504150912:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762114.238303:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762114.238319:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2040 00000100:00080000:0.0:1789762114.238322:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762114.238324:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762114.238328:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762114.238339:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504151168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762114.238351:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762114.238354:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762114.238356:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762114.238359:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2040 00000100:00080000:0.0:1789762114.238379:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762114.238383:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762114.238388:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789762114.238408:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762114.238419:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504151296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762114.238433:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762114.238437:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762114.238439:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762114.238443:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2040 00000100:00080000:0.0:1789762114.238447:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762114.238458:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762114.238476:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762114/real 1789762114] req@ffff9018c8b9b100 x1876699504150912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789762114.238480:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762114.238504:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504151424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762114.238511:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504150912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762114.238521:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762114.238559:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762114.238561:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762114.238563:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762114.238574:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789762114.238580:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2040 00000100:00100000:1.0:1789762114.238591:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504150912:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762114.238592:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762114.238609:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762114.238616:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762114.238629:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504151552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762114.238634:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504151040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762114.238679:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762114.238714:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762114/real 1789762114] req@ffff9018c8b9b480 x1876699504151040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762114.238725:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504151040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762114.238741:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762114.238747:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762114.238752:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504151040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762114.238763:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504151168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762114.238789:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762114.238801:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762114/real 1789762114] req@ffff9018c8b98000 x1876699504151168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762114.238809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504151168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762114.238815:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762114.238817:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762114.238819:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504151168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762114.238823:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504151296:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762114.238836:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762114.238843:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762114/real 1789762114] req@ffff9018c8b9aa00 x1876699504151296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762114.238850:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504151296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762114.238856:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762114.238857:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762114.238859:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504151296:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762114.238863:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504151424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762114.238874:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762114.238880:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762114/real 1789762114] req@ffff9018c8b9b800 x1876699504151424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762114.238887:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504151424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762114.238892:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762114.238894:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762114.238896:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504151424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762114.238908:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504151552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762114.238919:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762114.238925:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762114/real 1789762114] req@ffff9018c8b9a680 x1876699504151552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762114.238932:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504151552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762114.238937:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762114.238939:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762114.238941:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504151552:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762114.878241:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762115.902272:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762116.926251:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762117.950232:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762118.974227:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762119.358195:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762119.358206:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762119.358211:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762119.358224:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2045 00000100:00080000:3.0:1789762119.358239:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762119.358248:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762119.358301:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504151680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762119.358328:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762119.358331:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762119.358332:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762119.358336:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2045 00000100:00080000:3.0:1789762119.358339:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762119.358342:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762119.358353:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762119.358365:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504151808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762119.358371:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504151680:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762119.358376:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762119.358380:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762119.358381:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762119.358385:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2045 00000100:00080000:3.0:1789762119.358387:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762119.358389:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762119.358392:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762119.358401:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504151936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762119.358407:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762119.358410:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762119.358411:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762119.358414:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2045 00000100:00080000:3.0:1789762119.358417:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762119.358419:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762119.358423:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762119.358431:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504152064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762119.358438:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762119.358441:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762119.358442:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762119.358445:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2045 00000100:00080000:3.0:1789762119.358459:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762119.358462:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762119.358465:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762119.358470:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762119.358476:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504152192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762119.358483:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762119.358485:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762119.358487:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762119.358490:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2045 00000100:00080000:3.0:1789762119.358492:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762119.358494:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762119.358497:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762119.358504:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504152320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762119.358529:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762119/real 1789762119] req@ffff9018d070f100 x1876699504151680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762119.358546:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504151680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762119.358555:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762119.358558:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762119.358561:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504151680:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762119.358585:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504151808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762119.358607:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762119.358621:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762119/real 1789762119] req@ffff9018d070f800 x1876699504151808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762119.358635:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504151808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762119.358644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762119.358648:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762119.358651:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504151808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762119.358675:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504151936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762119.358694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762119.358712:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762119/real 1789762119] req@ffff9018d070d500 x1876699504151936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762119.358720:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504151936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762119.358726:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762119.358728:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762119.358730:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504151936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762119.358734:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504152064:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762119.358746:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762119.358752:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762119/real 1789762119] req@ffff9018d070dc00 x1876699504152064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762119.358759:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504152064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762119.358764:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762119.358766:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762119.358768:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504152064:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762119.358772:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504152192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762119.358782:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762119.358788:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762119/real 1789762119] req@ffff9018d070c000 x1876699504152192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762119.358795:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504152192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762119.358800:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762119.358802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762119.358804:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504152192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762119.358807:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504152320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762119.358817:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762119.358822:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762119/real 1789762119] req@ffff9018d070e300 x1876699504152320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762119.358829:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504152320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762119.358834:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762119.358836:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762119.358837:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504152320:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762119.998196:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762121.022281:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762122.046296:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762123.070226:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762124.094285:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762124.478194:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762124.478222:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762124.478228:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762124.478241:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2051 00000100:00080000:3.0:1789762124.478283:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762124.478294:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762124.478347:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504152448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762124.478400:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762124.478405:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762124.478407:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762124.478430:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2051 00000100:00080000:3.0:1789762124.478436:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762124.478442:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762124.478456:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762124.478474:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504152448:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762124.478476:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504152576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762124.478507:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762124.478510:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762124.478512:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762124.478515:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2051 00000100:00080000:3.0:1789762124.478518:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762124.478521:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762124.478524:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762124.478532:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504152704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762124.478539:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762124.478541:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762124.478543:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762124.478562:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2051 00000100:00080000:3.0:1789762124.478567:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762124.478572:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762124.478579:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762124.478593:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504152832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762124.478603:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762124.478611:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762124.478612:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762124.478640:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762124.478646:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2051 00000100:00080000:3.0:1789762124.478651:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762124.478655:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762124.478718:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762124/real 1789762124] req@ffff9018d070ce00 x1876699504152448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762124.478740:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504152448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762124.478752:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762124.478757:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762124.478761:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504152448:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762124.478806:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504152576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762124.478833:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762124.478846:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762124/real 1789762124] req@ffff9018d070ca80 x1876699504152576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762124.478858:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504152576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762124.478872:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762124.478876:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762124.478879:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504152576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762124.478885:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504152704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762124.478903:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762124.478914:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762124/real 1789762124] req@ffff9018d070fb80 x1876699504152704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789762124.478924:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762124.478927:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504152704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762124.478938:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762124.478941:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762124.478947:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504152960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762124.478949:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504152704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762124.478958:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504152832:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762124.478983:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762124.478997:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762124/real 1789762124] req@ffff9018d070d180 x1876699504152832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762124.479022:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762124.479028:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762124.479031:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762124.479036:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504152832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762124.479086:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2051 00000100:00080000:0.0:1789762124.479087:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762124.479092:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762124.479094:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762124.479099:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762124.479102:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504152832:192.168.203.158@tcp:8:kworker.0 00000008:00100000:3.0:1789762124.479109:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762124.479143:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504153088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762124.479154:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504152960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762124.479202:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762124.479222:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762124/real 1789762124] req@ffff9018d070d880 x1876699504152960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762124.479236:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504152960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762124.479244:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762124.479247:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762124.479250:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504152960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762124.479262:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504153088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762124.479282:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762124.479291:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762124/real 1789762124] req@ffff9018d070e680 x1876699504153088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762124.479302:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504153088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762124.479313:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762124.479318:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762124.479322:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504153088:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762125.118186:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762126.142284:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762127.166165:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762128.190250:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762129.214170:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762129.598181:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762129.598191:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762129.598195:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762129.598205:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2056 00000100:00080000:0.0:1789762129.598215:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762129.598221:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762129.598256:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504153216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762129.598284:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762129.598287:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762129.598289:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762129.598292:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2056 00000100:00080000:0.0:1789762129.598295:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762129.598298:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762129.598306:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762129.598322:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504153344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762129.598337:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762129.598343:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504153216:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762129.598344:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762129.598349:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762129.598359:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2056 00000100:00080000:0.0:1789762129.598365:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762129.598370:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762129.598377:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762129.598390:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504153472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762129.598399:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762129.598402:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762129.598404:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762129.598408:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2056 00000100:00080000:0.0:1789762129.598412:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762129.598416:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762129.598420:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789762129.598428:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762129.598433:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504153600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762129.598443:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762129.598447:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762129.598449:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762129.598452:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2056 00000100:00080000:0.0:1789762129.598454:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762129.598458:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762129.598462:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762129.598473:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762129/real 1789762129] req@ffff9018c8b98380 x1876699504153216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762129.598474:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504153728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762129.598515:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762129.598518:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762129.598520:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762129.598523:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504153216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762129.598528:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2056 00000100:00080000:0.0:1789762129.598533:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762129.598542:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762129.598549:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762129.598557:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789762129.598563:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504153856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762129.598564:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762129.598574:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504153216:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762129.598628:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504153344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762129.598650:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762129.598661:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762129/real 1789762129] req@ffff9018c8b99c00 x1876699504153344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762129.598692:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504153344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762129.598701:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762129.598705:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762129.598709:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504153344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762129.598715:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504153472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762129.598733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762129.598743:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762129/real 1789762129] req@ffff9018c8b99500 x1876699504153472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762129.598772:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504153472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762129.598782:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762129.598786:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762129.598789:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504153472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762129.598797:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504153600:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762129.598817:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762129.598827:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762129/real 1789762129] req@ffff9018c8b9ad80 x1876699504153600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762129.598856:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504153600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762129.598863:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762129.598865:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762129.598868:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504153600:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762129.598872:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504153728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762129.598885:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762129.598892:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762129/real 1789762129] req@ffff9018c8b9a300 x1876699504153728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762129.598901:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504153728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762129.598908:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762129.598910:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762129.598912:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504153728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762129.598916:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504153856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762129.598932:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762129.598944:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762129/real 1789762129] req@ffff9018c8b9a680 x1876699504153856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762129.598961:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504153856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762129.598972:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762129.598976:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762129.598980:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504153856:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762130.238236:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762131.262208:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762132.286189:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762133.310228:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762134.334191:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762134.718145:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762134.718156:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762134.718160:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762134.718169:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2061 00000100:00080000:0.0:1789762134.718178:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762134.718184:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762134.718216:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504153984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762134.718253:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762134.718257:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762134.718258:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762134.718261:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2061 00000100:00080000:0.0:1789762134.718264:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762134.718266:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762134.718274:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762134.718288:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504154112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762134.718319:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504153984:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762134.718329:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762134.718334:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762134.718336:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762134.718342:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2061 00000100:00080000:0.0:1789762134.718346:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762134.718350:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762134.718355:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762134.718386:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504154240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762134.718393:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762134.718395:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762134.718396:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762134.718399:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2061 00000100:00080000:0.0:1789762134.718402:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762134.718404:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762134.718408:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762134.718413:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762134.718422:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504154368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762134.718445:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762134.718448:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762134.718449:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762134.718453:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2061 00000100:00100000:1.0:1789762134.718457:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762134/real 1789762134] req@ffff9018c8b9b800 x1876699504153984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762134.718460:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762134.718470:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762134.718476:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762134.718497:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504153984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762134.718498:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504154496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762134.718534:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762134.718536:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762134.718540:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789762134.718553:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762134.718557:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762134.718559:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504153984:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762134.718564:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2061 00000100:00080000:0.0:1789762134.718571:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762134.718576:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762134.718582:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762134.718589:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504154112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762134.718594:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504154624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762134.718614:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762134.718626:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762134/real 1789762134] req@ffff9018c8b9aa00 x1876699504154112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762134.718634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504154112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762134.718645:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762134.718647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762134.718650:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504154112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762134.718653:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504154240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762134.718664:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762134.718670:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762134/real 1789762134] req@ffff9018c8b98000 x1876699504154240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762134.718677:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504154240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762134.718683:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762134.718684:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762134.718686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504154240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762134.718689:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504154368:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762134.718700:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762134.718705:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762134/real 1789762134] req@ffff9018c8b9b480 x1876699504154368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762134.718713:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504154368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762134.718718:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762134.718719:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762134.718729:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504154368:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762134.718736:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504154496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762134.718747:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762134.718754:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762134/real 1789762134] req@ffff9018c8b9b100 x1876699504154496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762134.718761:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504154496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762134.718766:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762134.718768:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762134.718771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504154496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762134.718781:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504154624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762134.718791:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762134.718797:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762134/real 1789762134] req@ffff9018c8b98e00 x1876699504154624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762134.718804:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504154624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762134.718809:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762134.718811:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762134.718812:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504154624:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762135.358210:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762136.382157:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762137.406217:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762138.430197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762139.454162:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762139.838151:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762139.838159:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762139.838163:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762139.838170:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2066 00000100:00080000:2.0:1789762139.838180:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762139.838186:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762139.838219:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504154752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762139.838243:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762139.838246:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762139.838247:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762139.838250:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2066 00000100:00080000:2.0:1789762139.838253:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762139.838255:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762139.838262:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762139.838273:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504154880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762139.838281:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762139.838284:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762139.838286:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762139.838288:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2066 00000100:00080000:2.0:1789762139.838291:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762139.838296:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762139.838299:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762139.838301:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504154752:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762139.838306:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504155008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762139.838316:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762139.838318:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762139.838320:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762139.838323:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2066 00000100:00080000:2.0:1789762139.838325:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762139.838327:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762139.838330:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762139.838339:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504155136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762139.838346:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762139.838348:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762139.838350:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762139.838353:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2066 00000100:00080000:2.0:1789762139.838368:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762139.838371:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789762139.838376:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789762139.838378:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762139.838388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504155264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762139.838396:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762139.838398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762139.838400:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762139.838402:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2066 00000100:00080000:2.0:1789762139.838404:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762139.838407:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762139.838410:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762139.838419:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504155392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762139.838426:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762139/real 1789762139] req@ffff9018d0f24380 x1876699504154752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762139.838444:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504154752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762139.838453:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762139.838456:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762139.838460:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504154752:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762139.838466:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504154880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762139.838482:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762139.838491:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762139/real 1789762139] req@ffff9018d0f27800 x1876699504154880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762139.838500:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504154880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762139.838520:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762139.838522:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762139.838525:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504154880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762139.838544:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504155008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762139.838572:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762139.838579:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762139/real 1789762139] req@ffff9018d0f25180 x1876699504155008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762139.838587:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504155008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762139.838598:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762139.838600:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762139.838603:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504155008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762139.838606:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504155136:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762139.838617:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762139.838622:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762139/real 1789762139] req@ffff9018d0f24a80 x1876699504155136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762139.838630:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504155136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762139.838636:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762139.838638:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762139.838640:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504155136:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762139.838643:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504155264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762139.838653:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762139.838659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762139/real 1789762139] req@ffff9018d0f25500 x1876699504155264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762139.838667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504155264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762139.838672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762139.838674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762139.838677:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504155264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762139.838681:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504155392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762139.838691:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762139.838696:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762139/real 1789762139] req@ffff9018d0f27480 x1876699504155392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762139.838703:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504155392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762139.838709:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762139.838711:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762139.838712:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504155392:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762140.478183:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762141.502208:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762142.526179:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762143.550202:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762144.574204:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762144.958130:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762144.958141:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762144.958145:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762144.958155:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2071 00000100:00080000:1.0:1789762144.958169:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762144.958183:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762144.958237:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504155520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762144.958268:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762144.958270:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762144.958272:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762144.958276:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2071 00000100:00080000:1.0:1789762144.958279:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762144.958283:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762144.958293:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762144.958304:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504155648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762144.958312:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762144.958314:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762144.958316:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762144.958319:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2071 00000100:00080000:1.0:1789762144.958321:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762144.958323:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762144.958326:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762144.958332:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504155520:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762144.958336:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504155776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762144.958348:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762144.958350:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762144.958351:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762144.958354:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2071 00000100:00080000:1.0:1789762144.958356:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762144.958358:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762144.958362:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762144.958370:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504155904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762144.958377:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762144.958380:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762144.958382:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762144.958384:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2071 00000100:00080000:1.0:1789762144.958400:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762144.958403:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762144.958406:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762144.958414:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504156032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762144.958418:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762144.958427:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762144.958430:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762144.958431:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762144.958436:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2071 00000100:00080000:1.0:1789762144.958438:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762144.958441:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762144.958444:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762144.958452:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504156160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762144.958483:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762144/real 1789762144] req@ffff9018c66a5180 x1876699504155520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762144.958500:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504155520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762144.958515:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762144.958518:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762144.958521:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504155520:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762144.958527:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504155648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762144.958547:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762144.958557:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762144/real 1789762144] req@ffff9018c66a6680 x1876699504155648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762144.958565:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504155648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762144.958571:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762144.958573:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762144.958575:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504155648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762144.958606:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504155776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762144.958619:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762144.958625:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762144/real 1789762144] req@ffff9018c66a4700 x1876699504155776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762144.958633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504155776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762144.958639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762144.958640:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762144.958642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504155776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762144.958647:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504155904:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762144.958659:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762144.958665:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762144/real 1789762144] req@ffff9018c66a6300 x1876699504155904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762144.958672:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504155904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762144.958678:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762144.958680:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762144.958681:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504155904:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762144.958685:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504156032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762144.958694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762144.958701:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762144/real 1789762144] req@ffff9018c66a4e00 x1876699504156032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762144.958708:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504156032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762144.958713:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762144.958715:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762144.958717:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504156032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762144.958720:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504156160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762144.958731:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762144.958736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762144/real 1789762144] req@ffff9018c66a5500 x1876699504156160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762144.958744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504156160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762144.958749:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762144.958751:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762144.958752:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504156160:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762145.598276:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762146.622172:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762147.646212:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762148.670196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762149.694198:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762150.078113:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762150.078121:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762150.078124:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762150.078133:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2076 00000100:00080000:2.0:1789762150.078142:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762150.078149:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762150.078179:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504156288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762150.078202:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762150.078205:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762150.078207:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762150.078210:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2076 00000100:00080000:2.0:1789762150.078212:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762150.078214:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762150.078224:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762150.078239:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504156416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762150.078250:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504156288:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762150.078252:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762150.078258:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762150.078260:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762150.078265:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2076 00000100:00080000:2.0:1789762150.078269:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762150.078273:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762150.078279:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762150.078291:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504156544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762150.078301:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762150.078304:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762150.078306:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762150.078309:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762150.078313:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2076 00000100:00080000:2.0:1789762150.078319:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762150.078323:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762150.078329:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762150.078341:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504156672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762150.078346:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762150/real 1789762150] req@ffff9018d0f24e00 x1876699504156288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762150.078365:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762150.078370:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789762150.078371:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504156288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762150.078373:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762150.078385:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2076 00000100:00080000:0.0:1789762150.078388:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762150.078391:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762150.078393:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762150.078401:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504156288:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762150.078402:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762150.078411:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762150.078425:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504156800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762150.078442:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504156416:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789762150.078459:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762150.078465:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762150.078468:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762150.078469:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762150.078477:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2076 00000100:00080000:2.0:1789762150.078483:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762150.078486:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762150/real 1789762150] req@ffff9018d0f24000 x1876699504156416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762150.078489:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762150.078497:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762150.078510:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504156416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762150.078513:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504156928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762150.078543:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762150.078549:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762150.078552:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504156416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762150.078557:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504156544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762150.078574:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762150.078585:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762150/real 1789762150] req@ffff9018d0f24700 x1876699504156544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762150.078598:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504156544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762150.078606:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762150.078610:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762150.078613:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504156544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762150.078620:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504156672:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762150.078640:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762150.078649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762150/real 1789762150] req@ffff9018d0f26680 x1876699504156672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762150.078657:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504156672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762150.078662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762150.078664:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762150.078667:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504156672:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762150.078679:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504156800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762150.078691:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762150.078697:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762150/real 1789762150] req@ffff9018d0f27480 x1876699504156800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762150.078704:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504156800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762150.078709:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762150.078711:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762150.078713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504156800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762150.078716:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504156928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762150.078726:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762150.078731:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762150/real 1789762150] req@ffff9018d0f25500 x1876699504156928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762150.078739:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504156928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762150.078743:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762150.078745:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762150.078747:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504156928:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762150.718201:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762151.742174:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762152.766224:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762153.790203:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762154.814184:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762155.198161:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762155.198172:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762155.198177:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762155.198187:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2081 00000100:00080000:0.0:1789762155.198197:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762155.198204:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762155.198269:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504157056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762155.198299:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762155.198303:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762155.198305:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762155.198309:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2081 00000100:00080000:0.0:1789762155.198313:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762155.198317:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762155.198327:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762155.198340:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504157184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762155.198351:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762155.198355:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762155.198359:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762155.198360:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504157056:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762155.198365:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2081 00000100:00080000:0.0:1789762155.198372:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762155.198376:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762155.198381:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762155.198412:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504157312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762155.198422:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762155.198426:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762155.198428:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762155.198432:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2081 00000800:00000100:1.0:1789762155.198437:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762155.198438:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762155.198445:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762155.198452:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762155.198464:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504157440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762155.198475:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762155.198479:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762155.198483:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762155.198484:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762155/real 1789762155] req@ffff9018c8b99f80 x1876699504157056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762155.198489:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2081 00000100:00080000:0.0:1789762155.198496:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762155.198503:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762155.198508:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762155.198511:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504157056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762155.198522:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504157568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762155.198528:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762155.198537:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789762155.198542:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762155.198543:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504157056:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762155.198551:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762155.198555:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762155.198560:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2081 00000100:00080000:0.0:1789762155.198564:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762155.198568:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762155.198570:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504157184:192.168.203.158@tcp:38:kworker.0 00000008:00100000:0.0:1789762155.198576:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762155.198592:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504157696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762155.198598:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762155.198612:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762155/real 1789762155] req@ffff9018c8b98700 x1876699504157184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762155.198630:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504157184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762155.198637:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762155.198639:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762155.198642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504157184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762155.198647:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504157312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762155.198659:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762155.198666:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762155/real 1789762155] req@ffff9018c8b9bb80 x1876699504157312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762155.198673:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504157312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762155.198679:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762155.198682:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762155.198686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504157312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762155.198692:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504157440:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762155.198712:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762155.198724:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762155/real 1789762155] req@ffff9018c8b99880 x1876699504157440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762155.198736:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504157440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762155.198742:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762155.198745:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762155.198747:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504157440:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762155.198751:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504157568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762155.198766:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762155.198773:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762155/real 1789762155] req@ffff9018c8b98a80 x1876699504157568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762155.198780:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504157568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762155.198786:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762155.198788:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762155.198790:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504157568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762155.198802:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504157696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762155.198812:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762155.198818:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762155/real 1789762155] req@ffff9018c8b99f80 x1876699504157696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762155.198831:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504157696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762155.198840:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762155.198843:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762155.198846:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504157696:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762155.838218:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762156.862199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762157.886215:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762158.910196:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762159.934200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762160.318178:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762160.318191:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762160.318197:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762160.318209:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2086 00000100:00080000:3.0:1789762160.318223:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762160.318233:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762160.318281:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504157824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762160.318319:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762160.318323:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762160.318326:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762160.318334:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2086 00000100:00080000:3.0:1789762160.318339:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762160.318347:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762160.318374:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762160.318384:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504157824:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762160.318394:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504157952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762160.318409:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762160.318414:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762160.318416:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762160.318420:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2086 00000100:00080000:3.0:1789762160.318424:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762160.318428:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762160.318433:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762160.318445:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504158080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762160.318512:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504157952:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762160.318576:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762160.318581:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762160.318584:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762160.318590:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2086 00000100:00080000:3.0:1789762160.318596:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762160.318601:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762160.318608:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762160.318618:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504158080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762160.318627:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504158208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762160.318642:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762160.318666:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504158208:192.168.203.158@tcp:8:kworker.0 00000100:00080000:3.0:1789762160.319997:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762160.320004:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762160.320014:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2086 00000100:00080000:3.0:1789762160.320068:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762160.320076:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762160.320086:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762160.320108:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504158336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762160.320129:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762160.320133:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762160.320135:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762160.320140:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2086 00000100:00080000:3.0:1789762160.320144:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762160.320150:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762160.320156:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762160.320171:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504158464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762160.321012:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504158336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762160.321162:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504158464:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762160.958196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762161.982169:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762163.006195:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762163.326133:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762163.326144:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789762163.326147:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762163.326149:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789762163.326151:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762163.326153:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789762163.326155:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762163.326157:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789762163.326159:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762163.326161:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789762163.326163:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762163.326165:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000400:00000100:2.0:1789762163.390343:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789762163.390391:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762163.390410:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762163.390458:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762163.390470:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762163.390483:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762163.390492:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762163.390505:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789762163.390532:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762160/real 1789762163] req@ffff9018d070d880 x1876699504158336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762163.390565:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504158336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762163.390579:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762163.390585:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762163.390591:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504158336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762163.390601:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762160/real 1789762163] req@ffff9018d070d180 x1876699504158464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762163.390616:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504158464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762163.390639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762163.390644:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762163.390648:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504158464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762163.390655:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762160/real 1789762163] req@ffff9018d070e680 x1876699504158208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762163.390670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504158208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762163.390678:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762163.390682:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762163.390686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504158208:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762163.390693:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762160/real 1789762163] req@ffff9018d070df80 x1876699504158080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762163.390706:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504158080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762163.390722:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762163.390727:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762163.390730:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504158080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762163.390738:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762160/real 1789762163] req@ffff9018d070ed80 x1876699504157952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762163.390751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504157952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762163.390759:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762163.390763:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762163.390766:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504157952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762163.390777:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762160/real 1789762163] req@ffff9018d070f480 x1876699504157824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762163.390791:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504157824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762163.390799:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762163.390803:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762163.390806:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504157824:192.168.203.158@tcp:250:kworker.0 00010000:00010000:2.0:1789762164.030161:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762164.350175:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762164.350186:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762164.350191:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762164.350204:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2091 00000100:00080000:2.0:1789762164.350216:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762164.350224:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762164.350272:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504158592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762164.350301:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762164.350305:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762164.350306:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762164.350309:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2091 00000100:00080000:2.0:1789762164.350312:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762164.350315:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762164.350340:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762164.350352:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504158720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762164.350360:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762164.350365:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762164.350367:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762164.350372:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2091 00000100:00080000:2.0:1789762164.350379:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762164.350384:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762164.350389:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762164.350403:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504158848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762164.350417:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762164.350425:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762164.350429:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762164.350431:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504158592:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762164.350442:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2091 00000100:00080000:2.0:1789762164.350450:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762164.350458:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762164.350466:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762164.350480:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504158976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762164.350488:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762164.350491:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762164.350493:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762164.350496:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2092 00000100:00080000:2.0:1789762164.350499:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762164.350503:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762164.350508:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762164.350519:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504159104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762164.350527:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762164.350532:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762164.350534:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762164.350536:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2092 00000100:00080000:2.0:1789762164.350539:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762164.350541:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762164.350548:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762164.350549:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762164.350564:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504159232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762164.350619:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762164/real 1789762164] req@ffff9018d0f24a80 x1876699504158592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762164.350641:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504158592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762164.350655:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762164.350660:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762164.350664:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504158592:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762164.350676:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504158720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762164.350720:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762164.350743:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762164/real 1789762164] req@ffff9018d0f25180 x1876699504158720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762164.350766:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504158720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762164.350776:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762164.350780:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762164.350784:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504158720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762164.350815:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504158848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762164.350840:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762164.350852:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762164/real 1789762164] req@ffff9018d0f27800 x1876699504158848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762164.350865:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504158848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762164.350886:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762164.350890:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762164.350898:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504158848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762164.350906:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504158976:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762164.350927:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762164.350944:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762164/real 1789762164] req@ffff9018d0f24380 x1876699504158976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762164.350975:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504158976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762164.350990:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762164.350995:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762164.351001:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504158976:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762164.351011:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504159104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762164.351108:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762164.351124:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762164/real 1789762164] req@ffff9018d0f26a00 x1876699504159104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762164.351138:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504159104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762164.351147:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762164.351150:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762164.351154:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504159104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762164.351160:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504159232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762164.351181:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762164.351190:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762164/real 1789762164] req@ffff9018d0f25c00 x1876699504159232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762164.351202:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504159232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762164.351217:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762164.351221:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762164.351224:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504159232:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762165.054276:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762166.078188:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762167.102292:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762168.126278:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762169.150402:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762169.534167:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762169.534182:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762169.534187:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762169.534201:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2096 00000100:00080000:0.0:1789762169.534215:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762169.534224:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762169.534297:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504159360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762169.534331:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762169.534336:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762169.534338:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762169.534343:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2096 00000100:00080000:0.0:1789762169.534348:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762169.534354:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762169.534367:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762169.534376:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504159360:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762169.534385:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504159488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762169.534398:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762169.534403:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762169.534405:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762169.534409:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2096 00000100:00080000:0.0:1789762169.534413:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762169.534417:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762169.534422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762169.534444:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504159616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762169.534454:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762169.534458:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762169.534460:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762169.534464:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2096 00000100:00080000:0.0:1789762169.534468:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762169.534474:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789762169.534475:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789762169.534482:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762169.534497:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504159744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762169.534508:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762169.534512:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762169.534514:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762169.534519:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2096 00000100:00080000:0.0:1789762169.534523:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762169.534528:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762169.534535:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762169/real 1789762169] req@ffff9018c8b98e00 x1876699504159360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789762169.534536:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762169.534566:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504159872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762169.534575:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504159360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762169.534582:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762169.534592:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762169.534595:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762169.534596:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762169.534602:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2096 00000100:00080000:2.0:1789762169.534606:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789762169.534609:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762169.534614:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504159360:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762169.534620:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762169.534628:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762169.534642:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504160000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762169.534652:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504159488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762169.534706:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762169.534720:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762169/real 1789762169] req@ffff9018c8b9b100 x1876699504159488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762169.534744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504159488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762169.534752:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762169.534756:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762169.534760:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504159488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762169.534766:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504159616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762169.534787:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762169.534797:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762169/real 1789762169] req@ffff9018c8b9b480 x1876699504159616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762169.534804:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504159616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762169.534810:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762169.534812:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762169.534814:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504159616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762169.534818:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504159744:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762169.534830:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762169.534837:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762169/real 1789762169] req@ffff9018c8b98000 x1876699504159744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762169.534849:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504159744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762169.534854:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762169.534856:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762169.534858:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504159744:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762169.534861:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504159872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762169.534872:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762169.534877:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762169/real 1789762169] req@ffff9018c8b9aa00 x1876699504159872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762169.534884:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504159872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762169.534889:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762169.534891:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762169.534893:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504159872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762169.534906:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504160000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762169.534917:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762169.534923:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762169/real 1789762169] req@ffff9018c8b9b800 x1876699504160000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762169.534929:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504160000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762169.534938:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762169.534940:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762169.534942:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504160000:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762170.174260:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762171.198291:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762172.222186:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762173.246202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762174.270292:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762174.654318:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762174.654328:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762174.654334:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762174.654346:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2101 00000100:00080000:3.0:1789762174.654360:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762174.654370:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762174.654429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504160128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762174.654462:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762174.654467:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762174.654469:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762174.654476:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2101 00000100:00080000:3.0:1789762174.654481:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762174.654487:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762174.654501:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762174.654525:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504160128:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762174.654528:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504160256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762174.654552:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762174.654559:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762174.654563:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762174.654573:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2101 00000100:00080000:3.0:1789762174.654590:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762174.654613:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789762174.654622:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789762174.654626:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762174.654665:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504160384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762174.654679:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762174.654689:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762174/real 1789762174] req@ffff9018d070fb80 x1876699504160128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762174.654712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504160128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762174.654725:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762174.654729:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762174.654733:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504160128:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762174.654761:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504160256:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762174.654772:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762174.654775:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762174.654782:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2101 00000100:00080000:3.0:1789762174.654789:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762174.654791:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762174.654803:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762174.654814:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762174.654815:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762174/real 1789762174] req@ffff9018d070ca80 x1876699504160256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762174.654841:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504160512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762174.654843:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504160256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762174.654872:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762174.654875:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762174.654878:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504160256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762174.654885:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504160384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762174.654906:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762174.654916:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762174/real 1789762174] req@ffff9018d070ce00 x1876699504160384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762174.654929:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504160384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762174.654957:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762174.654958:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762174.654970:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762174.654974:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762174.654976:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762174.654991:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504160384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762174.655026:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504160512:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762174.655110:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762174.655130:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762174/real 1789762174] req@ffff9018d070e300 x1876699504160512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762174.655153:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504160512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762174.655163:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762174.655167:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762174.655171:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504160512:192.168.203.158@tcp:8:kworker.0 00000100:00080000:3.0:1789762174.655184:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2101 00000100:00080000:3.0:1789762174.655192:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762174.655199:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762174.655207:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762174.655225:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504160640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762174.655338:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762174.655354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762174.655358:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762174.655366:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2101 00000100:00080000:3.0:1789762174.655378:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762174.655382:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504160640:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762174.655392:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762174.655403:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762174.655433:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504160768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762174.655445:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762174.655478:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762174/real 1789762174] req@ffff9018d070c000 x1876699504160640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762174.655503:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504160640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762174.655515:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762174.655520:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762174.655531:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504160640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762174.655548:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504160768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762174.655567:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762174.655584:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762174/real 1789762174] req@ffff9018d070dc00 x1876699504160768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762174.655596:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504160768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762174.655611:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762174.655615:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762174.655618:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504160768:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762175.294348:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762176.318335:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762177.342161:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762178.366180:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762179.390233:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762179.774182:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762179.774193:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762179.774197:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762179.774225:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2106 00000100:00080000:0.0:1789762179.774236:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762179.774243:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762179.774280:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504160896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762179.774313:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762179.774317:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762179.774319:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762179.774324:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2106 00000100:00080000:0.0:1789762179.774329:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762179.774335:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762179.774345:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762179.774361:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504160896:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762179.774362:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504161024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762179.774383:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762179.774389:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762179.774391:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762179.774399:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2106 00000100:00080000:0.0:1789762179.774403:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762179.774407:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762179.774413:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762179.774428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504161152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762179.774439:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762179.774443:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762179.774445:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762179.774449:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2106 00000100:00080000:0.0:1789762179.774455:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762179.774462:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762179.774463:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762179.774473:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762179.774489:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504161280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762179.774501:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762179.774505:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762179.774507:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762179.774513:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762179/real 1789762179] req@ffff9018c8b9a680 x1876699504160896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762179.774514:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2106 00000100:00080000:0.0:1789762179.774527:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762179.774538:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762179.774542:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504160896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789762179.774549:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762179.774555:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762179.774560:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762179.774565:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504160896:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762179.774570:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504161408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762179.774583:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762179.774586:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762179.774589:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762179.774593:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2106 00000100:00080000:0.0:1789762179.774598:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762179.774599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504161024:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789762179.774612:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762179.774620:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762179.774626:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762179.774639:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504161536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762179.774646:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762179/real 1789762179] req@ffff9018c8b9a300 x1876699504161024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762179.774686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504161024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762179.774705:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762179.774708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762179.774711:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504161024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762179.774716:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504161152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762179.774734:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762179.774742:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762179/real 1789762179] req@ffff9018c8b9ad80 x1876699504161152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762179.774749:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504161152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762179.774754:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762179.774756:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762179.774758:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504161152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762179.774762:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504161280:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762179.774773:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762179.774779:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762179/real 1789762179] req@ffff9018c8b99500 x1876699504161280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762179.774787:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504161280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762179.774792:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762179.774794:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762179.774805:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504161280:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762179.774809:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504161408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762179.774824:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762179.774831:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762179/real 1789762179] req@ffff9018c8b99c00 x1876699504161408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762179.774838:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504161408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762179.774843:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762179.774845:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762179.774847:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504161408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762179.774861:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504161536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762179.774872:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762179.774878:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762179/real 1789762179] req@ffff9018c8b98380 x1876699504161536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762179.774885:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504161536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762179.774890:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762179.774891:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762179.774893:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504161536:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762180.414197:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762181.438315:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762182.462298:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762183.486295:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762184.510220:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762184.894138:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762184.894150:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762184.894155:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762184.894176:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2111 00000100:00080000:0.0:1789762184.894189:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762184.894195:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762184.894253:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504161664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762184.894279:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762184.894282:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762184.894284:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762184.894288:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2111 00000100:00080000:0.0:1789762184.894291:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762184.894298:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762184.894308:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762184.894319:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504161792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762184.894326:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504161664:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762184.894330:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762184.894334:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762184.894335:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762184.894339:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2111 00000100:00080000:0.0:1789762184.894341:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762184.894344:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762184.894347:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762184.894356:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504161920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762184.894363:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762184.894366:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762184.894368:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762184.894371:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2111 00000100:00080000:0.0:1789762184.894373:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762184.894376:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762184.894380:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762184.894388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504162048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762184.894395:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762184.894398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762184.894401:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762184.894403:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762184.894407:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2111 00000100:00080000:0.0:1789762184.894410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762184.894413:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762184.894417:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762184.894425:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504162176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762184.894432:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762184.894435:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762184.894436:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762184.894439:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2111 00000100:00080000:0.0:1789762184.894442:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762184.894445:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762184.894448:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762184.894455:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762184/real 1789762184] req@ffff9018c8b98380 x1876699504161664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762184.894457:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504162304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762184.894491:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504161664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762184.894503:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762184.894506:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762184.894510:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504161664:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762184.894534:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504161792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762184.894554:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762184.894564:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762184/real 1789762184] req@ffff9018c8b99c00 x1876699504161792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762184.894573:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504161792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762184.894578:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762184.894580:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762184.894583:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504161792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762184.894586:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504161920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762184.894598:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762184.894603:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762184/real 1789762184] req@ffff9018c8b99500 x1876699504161920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762184.894615:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504161920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762184.894621:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762184.894622:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762184.894624:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504161920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762184.894628:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504162048:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762184.894639:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762184.894645:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762184/real 1789762184] req@ffff9018c8b9ad80 x1876699504162048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762184.894652:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504162048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762184.894657:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762184.894659:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762184.894661:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504162048:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762184.894664:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504162176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762184.894675:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762184.894681:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762184/real 1789762184] req@ffff9018c8b9a300 x1876699504162176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762184.894687:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504162176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762184.894696:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762184.894698:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762184.894700:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504162176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762184.894704:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504162304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762184.894714:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762184.894719:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762184/real 1789762184] req@ffff9018c8b9a680 x1876699504162304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762184.894726:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504162304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762184.894731:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762184.894732:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762184.894734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504162304:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762185.534200:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762186.558280:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762187.582189:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762188.606268:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762189.630229:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762190.014219:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762190.014229:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762190.014233:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762190.014244:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2116 00000100:00080000:1.0:1789762190.014255:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762190.014262:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762190.014306:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504162432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762190.014332:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762190.014335:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762190.014337:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762190.014340:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2116 00000100:00080000:1.0:1789762190.014342:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762190.014345:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762190.014355:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762190.014364:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504162560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762190.014372:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762190.014375:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762190.014376:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762190.014381:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2116 00000100:00100000:2.0:1789762190.014382:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504162432:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762190.014384:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762190.014390:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762190.014394:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762190.014400:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504162688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762190.014406:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762190.014409:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762190.014410:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762190.014412:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2116 00000100:00080000:1.0:1789762190.014414:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762190.014417:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762190.014420:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762190.014428:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504162816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762190.014434:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762190.014437:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762190.014438:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762190.014441:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2116 00000100:00080000:1.0:1789762190.014443:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762190.014447:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762190.014450:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762190.014458:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504162944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762190.014461:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762190.014469:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762190.014472:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762190.014474:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762190.014478:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2116 00000100:00080000:1.0:1789762190.014480:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762190.014483:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762190.014486:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762190.014492:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504163072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762190.014509:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762190/real 1789762190] req@ffff9018c66a7800 x1876699504162432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762190.014525:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504162432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762190.014534:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762190.014537:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762190.014541:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504162432:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762190.014562:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504162560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762190.014578:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762190.014595:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762190/real 1789762190] req@ffff9018c66a7b80 x1876699504162560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762190.014607:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504162560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762190.014613:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762190.014615:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762190.014617:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504162560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762190.014621:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504162688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762190.014634:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762190.014640:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762190/real 1789762190] req@ffff9018c66a4000 x1876699504162688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762190.014647:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504162688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762190.014652:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762190.014654:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762190.014656:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504162688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762190.014659:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504162816:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762190.014671:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762190.014677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762190/real 1789762190] req@ffff9018c66a5f80 x1876699504162816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762190.014684:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504162816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762190.014689:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762190.014691:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762190.014693:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504162816:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762190.014696:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504162944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762190.014706:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762190.014712:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762190/real 1789762190] req@ffff9018c66a6d80 x1876699504162944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762190.014719:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504162944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762190.014724:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762190.014726:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762190.014728:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504162944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762190.014731:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504163072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762190.014740:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762190.014745:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762190/real 1789762190] req@ffff9018c66a5c00 x1876699504163072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762190.014757:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504163072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762190.014761:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762190.014763:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762190.014765:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504163072:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762190.654257:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762191.678202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762192.702231:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762193.726228:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762194.750223:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762195.134178:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762195.134190:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762195.134196:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762195.134209:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2121 00000100:00080000:3.0:1789762195.134222:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762195.134231:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762195.134280:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504163200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762195.134315:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762195.134319:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762195.134321:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762195.134326:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2121 00000100:00080000:3.0:1789762195.134330:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762195.134336:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762195.134348:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762195.134361:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504163200:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762195.134367:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504163328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762195.134381:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762195.134384:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762195.134386:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762195.134391:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2121 00000100:00080000:3.0:1789762195.134396:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762195.134400:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762195.134405:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762195.134419:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504163456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762195.134430:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762195.134434:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762195.134437:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762195.134438:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762195.134446:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2121 00000100:00080000:3.0:1789762195.134452:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762195.134458:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762195.134464:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762195.134481:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504163584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762195.134494:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762195/real 1789762195] req@ffff9018d070d500 x1876699504163200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762195.134497:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762195.134506:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762195.134511:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762195.134522:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2121 00000100:00080000:3.0:1789762195.134527:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762195.134533:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504163200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762195.134536:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762195.134546:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762195.134552:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762195.134560:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762195.134563:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504163712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762195.134569:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504163200:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762195.134585:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762195.134589:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762195.134592:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762195.134597:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2121 00000100:00100000:2.0:1789762195.134610:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504163328:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762195.134612:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762195.134621:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762195.134627:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762195.134641:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762195.134645:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504163840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762195.134661:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762195/real 1789762195] req@ffff9018d070f800 x1876699504163328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762195.134687:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504163328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762195.134696:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762195.134700:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762195.134704:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504163328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762195.134710:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504163456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762195.134732:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762195.134743:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762195/real 1789762195] req@ffff9018d070f100 x1876699504163456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762195.134756:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504163456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762195.134764:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762195.134767:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762195.134779:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504163456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762195.134786:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504163584:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762195.134808:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762195.134820:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762195/real 1789762195] req@ffff9018d070dc00 x1876699504163584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762195.134833:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504163584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762195.134842:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762195.134845:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762195.134849:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504163584:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762195.134855:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504163712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762195.134873:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762195.134884:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762195/real 1789762195] req@ffff9018d070c000 x1876699504163712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762195.134896:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504163712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762195.134905:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762195.134908:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762195.134911:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504163712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762195.134931:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504163840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762195.134950:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762195.134960:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762195/real 1789762195] req@ffff9018d070e300 x1876699504163840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762195.134972:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504163840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762195.134981:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762195.134984:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762195.134987:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504163840:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762195.774248:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762196.798248:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762197.822214:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762198.846244:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762199.870200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762200.254150:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762200.254165:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762200.254170:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762200.254183:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2126 00000100:00080000:2.0:1789762200.254198:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762200.254206:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762200.254257:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504163968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762200.254295:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762200.254299:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762200.254301:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762200.254306:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2126 00000100:00080000:2.0:1789762200.254310:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762200.254314:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762200.254327:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762200.254343:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504164096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762200.254351:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504163968:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762200.254358:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762200.254363:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762200.254365:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762200.254370:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2126 00000100:00080000:2.0:1789762200.254373:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762200.254377:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762200.254383:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762200.254395:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504164224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762200.254405:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762200.254409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762200.254411:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762200.254415:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2126 00000100:00080000:2.0:1789762200.254420:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762200.254425:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789762200.254438:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789762200.254442:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762200.254458:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504164352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762200.254468:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762200.254472:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762200.254474:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762200.254479:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2126 00000100:00080000:2.0:1789762200.254483:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762200.254489:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762200.254494:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762200/real 1789762200] req@ffff9018d0f26d80 x1876699504163968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789762200.254497:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762200.254518:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504164480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762200.254519:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504163968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762200.254537:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762200.254547:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762200.254550:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762200.254553:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762200.254557:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2126 00000100:00080000:0.0:1789762200.254561:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789762200.254565:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762200.254566:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504163968:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762200.254574:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762200.254581:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762200.254594:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504164608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762200.254595:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504164096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762200.254625:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762200.254647:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762200/real 1789762200] req@ffff9018d0f25f80 x1876699504164096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762200.254657:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504164096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762200.254663:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762200.254666:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762200.254668:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504164096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762200.254672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504164224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762200.254685:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762200.254691:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762200/real 1789762200] req@ffff9018d0f27100 x1876699504164224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762200.254699:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504164224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762200.254705:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762200.254707:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762200.254709:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504164224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762200.254712:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504164352:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762200.254731:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762200.254744:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762200/real 1789762200] req@ffff9018d0f26300 x1876699504164352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762200.254757:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504164352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762200.254766:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762200.254770:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762200.254772:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504164352:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762200.254777:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504164480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762200.254793:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762200.254801:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762200/real 1789762200] req@ffff9018d0f25c00 x1876699504164480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762200.254809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504164480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762200.254819:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762200.254821:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762200.254824:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504164480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762200.254845:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504164608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762200.254857:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762200.254864:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762200/real 1789762200] req@ffff9018d0f26a00 x1876699504164608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762200.254871:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504164608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762200.254877:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762200.254879:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762200.254881:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504164608:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762200.894196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762201.918189:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762202.942179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762203.966218:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762204.990154:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762205.374120:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762205.374132:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762205.374136:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762205.374146:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2131 00000100:00080000:3.0:1789762205.374158:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762205.374166:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762205.374215:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504164736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762205.374242:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762205.374246:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762205.374248:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762205.374252:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2131 00000100:00080000:3.0:1789762205.374255:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762205.374260:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762205.374270:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762205.374275:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504164736:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762205.374285:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504164864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762205.374293:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762205.374296:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762205.374297:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762205.374300:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2131 00000100:00080000:3.0:1789762205.374302:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762205.374305:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762205.374310:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762205.374324:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504164992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762205.374335:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762205.374339:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762205.374341:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762205.374346:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2131 00000800:00000100:0.0:1789762205.374348:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762205.374354:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762205.374362:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762205.374368:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762205.374381:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504165120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762205.374390:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762205.374391:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762205/real 1789762205] req@ffff9018d070ce00 x1876699504164736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762205.374394:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762205.374397:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762205.374405:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2131 00000100:00080000:3.0:1789762205.374408:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762205.374411:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762205.374418:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762205.374430:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504164736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762205.374431:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504165248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762205.374465:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762205.374468:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762205.374468:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762205.374471:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762205.374476:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762205.374477:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2131 00000100:00080000:3.0:1789762205.374482:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762205.374486:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504164736:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762205.374489:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762205.374497:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762205.374504:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504165376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762205.374518:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504164864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762205.374539:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762205.374548:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762205/real 1789762205] req@ffff9018d070ca80 x1876699504164864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762205.374557:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504164864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762205.374563:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762205.374565:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762205.374568:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504164864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762205.374571:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504164992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762205.374582:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762205.374587:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762205/real 1789762205] req@ffff9018d070fb80 x1876699504164992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762205.374595:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504164992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762205.374600:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762205.374602:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762205.374604:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504164992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762205.374608:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504165120:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762205.374619:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762205.374625:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762205/real 1789762205] req@ffff9018d070f480 x1876699504165120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762205.374632:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504165120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762205.374638:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762205.374640:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762205.374641:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504165120:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762205.374645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504165248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762205.374655:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762205.374661:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762205/real 1789762205] req@ffff9018d070ed80 x1876699504165248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762205.374669:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504165248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762205.374673:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762205.374675:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762205.374677:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504165248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762205.374688:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504165376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762205.374698:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762205.374704:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762205/real 1789762205] req@ffff9018d070df80 x1876699504165376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762205.374712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504165376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762205.374717:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762205.374720:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762205.374722:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504165376:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762206.014252:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762207.038214:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762208.062255:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762209.086246:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762210.110198:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762210.494098:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762210.494106:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762210.494110:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762210.494118:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2137 00000100:00080000:3.0:1789762210.494127:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762210.494134:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762210.494163:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504165504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762210.494184:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762210.494186:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762210.494187:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762210.494191:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2137 00000100:00080000:3.0:1789762210.494193:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762210.494197:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762210.494205:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762210.494215:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504165632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762210.494223:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762210.494225:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762210.494227:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762210.494230:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2137 00000100:00080000:3.0:1789762210.494232:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762210.494237:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504165504:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762210.494238:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762210.494247:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762210.494259:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504165760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762210.494268:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762210.494271:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762210.494272:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762210.494284:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2137 00000100:00080000:3.0:1789762210.494292:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762210.494298:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789762210.494304:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789762210.494307:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762210.494324:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504165888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762210.494336:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762210.494339:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789762210.494346:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762210/real 1789762210] req@ffff9018d070e680 x1876699504165504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762210.494367:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504165504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762210.494381:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762210.494386:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762210.494391:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504165504:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762210.494399:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504165632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762210.494422:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762210.494435:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762210.494438:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762210/real 1789762210] req@ffff9018d070d180 x1876699504165632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762210.494449:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2137 00000100:00100000:2.0:1789762210.494459:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504165632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762210.494470:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762210.494475:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762210.494479:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504165632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762210.494500:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504165760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762210.494519:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762210.494528:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762210/real 1789762210] req@ffff9018d070d880 x1876699504165760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762210.494537:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504165760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762210.494544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762210.494548:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762210.494549:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762210.494555:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762210.494559:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504165760:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789762210.494562:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762210.494570:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504165888:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762210.494596:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762210.494614:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762210/real 1789762210] req@ffff9018d070df80 x1876699504165888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762210.494618:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504166016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762210.494641:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762210.494645:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504165888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762210.494645:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762210.494649:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762210.494656:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2137 00000100:00080000:3.0:1789762210.494662:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762210.494668:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789762210.494669:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789762210.494672:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762210.494675:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762210.494680:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504165888:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762210.494684:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504166144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762210.494696:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504166016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762210.494716:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762210.494726:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762210/real 1789762210] req@ffff9018d070ed80 x1876699504166016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762210.494734:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504166016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762210.494740:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762210.494742:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762210.494744:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504166016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762210.494751:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504166144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762210.494764:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762210.494773:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762210/real 1789762210] req@ffff9018d070f480 x1876699504166144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762210.494785:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504166144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762210.494793:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762210.494796:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762210.494800:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504166144:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762211.134212:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762212.158259:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762213.182202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762214.206210:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762215.230143:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762215.614139:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762215.614150:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762215.614155:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762215.614165:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2142 00000100:00080000:3.0:1789762215.614177:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762215.614184:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762215.614228:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504166272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762215.614254:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762215.614257:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762215.614259:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762215.614262:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2142 00000100:00080000:3.0:1789762215.614265:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762215.614269:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762215.614281:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762215.614292:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504166272:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762215.614292:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504166400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762215.614306:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762215.614309:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762215.614310:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762215.614313:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2142 00000100:00080000:3.0:1789762215.614314:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762215.614317:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762215.614320:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762215.614328:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504166528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762215.614334:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762215.614336:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762215.614337:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762215.614340:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2142 00000100:00080000:3.0:1789762215.614359:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762215.614362:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762215.614366:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789762215.614370:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762215.614376:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504166656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762215.614386:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762215.614388:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762215.614391:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762215.614394:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2142 00000100:00080000:3.0:1789762215.614396:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762215.614399:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762215.614402:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762215.614409:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504166784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762215.614415:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762215.614418:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762215/real 1789762215] req@ffff9018d070fb80 x1876699504166272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762215.614420:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762215.614424:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762215.614434:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2142 00000100:00080000:3.0:1789762215.614439:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762215.614444:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504166272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762215.614447:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762215.614458:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762215.614460:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762215.614466:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762215.614470:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504166272:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762215.614475:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504166912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762215.614495:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504166400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762215.614513:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762215.614522:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762215/real 1789762215] req@ffff9018d070ca80 x1876699504166400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762215.614530:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504166400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762215.614537:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762215.614540:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762215.614543:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504166400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762215.614550:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504166528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762215.614570:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762215.614581:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762215/real 1789762215] req@ffff9018d070ce00 x1876699504166528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762215.614599:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504166528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762215.614605:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762215.614607:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762215.614609:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504166528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762215.614613:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504166656:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762215.614634:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762215.614642:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762215/real 1789762215] req@ffff9018d070e300 x1876699504166656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762215.614649:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504166656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762215.614654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762215.614656:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762215.614658:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504166656:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762215.614661:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504166784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762215.614672:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762215.614677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762215/real 1789762215] req@ffff9018d070c000 x1876699504166784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762215.614684:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504166784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762215.614693:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762215.614695:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762215.614697:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504166784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762215.614701:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504166912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762215.614712:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762215.614717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762215/real 1789762215] req@ffff9018d070dc00 x1876699504166912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762215.614724:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504166912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762215.614736:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762215.614738:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762215.614740:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504166912:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762216.254200:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762217.278188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762218.302182:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762219.326253:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762220.350188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762220.734111:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762220.734119:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762220.734122:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762220.734131:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2147 00000100:00080000:3.0:1789762220.734142:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762220.734149:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762220.734184:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504167040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762220.734212:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762220.734216:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762220.734218:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762220.734223:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2147 00000100:00080000:3.0:1789762220.734227:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762220.734232:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762220.734243:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762220.734258:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504167168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762220.734273:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762220.734275:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504167040:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762220.734279:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762220.734283:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762220.734293:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2147 00000100:00080000:3.0:1789762220.734297:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762220.734299:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762220.734303:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762220.734316:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504167296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762220.734324:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762220.734327:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762220.734330:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762220.734336:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2147 00000100:00080000:3.0:1789762220.734354:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762220.734360:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789762220.734364:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789762220.734378:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762220.734409:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504167424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762220.734417:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762220/real 1789762220] req@ffff9018d070f100 x1876699504167040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762220.734448:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762220.734454:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504167040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762220.734456:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762220.734463:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762220.734478:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2147 00000100:00080000:2.0:1789762220.734480:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762220.734484:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762220.734488:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762220.734496:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504167040:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762220.734497:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762220.734512:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762220.734529:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504167552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762220.734535:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504167168:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762220.734559:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762220.734570:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762220.734573:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762220.734577:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762220.734581:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2147 00000100:00080000:3.0:1789762220.734585:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762220.734586:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762220/real 1789762220] req@ffff9018d070f800 x1876699504167168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762220.734591:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762220.734598:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762220.734613:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504167680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762220.734622:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504167168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762220.734636:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762220.734642:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762220.734649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504167168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762220.734658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504167296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762220.734682:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762220.734694:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762220/real 1789762220] req@ffff9018d070d500 x1876699504167296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762220.734705:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504167296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762220.734711:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762220.734713:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762220.734715:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504167296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762220.734719:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504167424:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762220.734734:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762220.734741:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762220/real 1789762220] req@ffff9018d070dc00 x1876699504167424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762220.734748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504167424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762220.734754:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762220.734756:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762220.734758:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504167424:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762220.734771:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504167552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762220.734782:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762220.734789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762220/real 1789762220] req@ffff9018d070c000 x1876699504167552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762220.734796:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504167552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762220.734801:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762220.734802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762220.734804:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504167552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762220.734808:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504167680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762220.734818:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762220.734824:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762220/real 1789762220] req@ffff9018d070e300 x1876699504167680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762220.734836:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504167680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762220.734840:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762220.734842:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762220.734844:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504167680:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762221.374237:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762222.398194:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762223.422181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762224.446167:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762225.470221:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762225.854162:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762225.854173:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762225.854177:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762225.854189:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2152 00000100:00080000:0.0:1789762225.854201:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762225.854208:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762225.854251:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504167808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762225.854277:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762225.854280:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762225.854281:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762225.854285:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2152 00000100:00080000:0.0:1789762225.854287:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762225.854290:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762225.854300:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762225.854309:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504167936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762225.854318:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762225.854321:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762225.854324:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762225.854325:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504167808:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762225.854328:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2152 00000100:00080000:0.0:1789762225.854333:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762225.854336:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762225.854340:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762225.854348:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504168064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762225.854354:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762225.854357:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762225.854359:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762225.854362:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2152 00000100:00080000:0.0:1789762225.854366:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762225.854370:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762225.854376:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762225.854388:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504168192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762225.854399:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762225.854403:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762225.854406:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762225.854410:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2152 00000100:00080000:0.0:1789762225.854423:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762225.854428:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762225.854434:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762225.854448:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504168320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762225.854460:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762225.854464:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762225.854467:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762225.854468:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504167936:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789762225.854476:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2152 00000100:00080000:0.0:1789762225.854482:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762225.854488:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762225.854494:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762225.854505:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504168064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762225.854513:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504168448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762225.854525:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504168192:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762225.854546:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504168320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762225.854565:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504168448:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762226.494155:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762227.518179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762228.542178:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762228.926381:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:1.0:1789762228.926445:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762228.926463:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789762228.926505:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789762228.926513:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789762228.926521:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789762228.926531:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789762228.926540:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:2.0:1789762228.926655:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762225/real 1789762228] req@ffff9018c8b98e00 x1876699504168448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762228.926686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504168448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762228.926700:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762228.926708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762228.926712:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504168448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762228.926721:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762225/real 1789762228] req@ffff9018c8b9aa00 x1876699504167936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762228.926736:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504167936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762228.926744:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762228.926748:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762228.926751:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504167936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762228.926758:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762225/real 1789762228] req@ffff9018c8b98000 x1876699504168064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762228.926769:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504168064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762228.926776:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762228.926780:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762228.926783:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504168064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762228.926790:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762225/real 1789762228] req@ffff9018c8b9b480 x1876699504168192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762228.926802:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504168192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762228.926808:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762228.926813:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762228.926816:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504168192:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762228.926822:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762225/real 1789762228] req@ffff9018c8b9b100 x1876699504168320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762228.926833:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504168320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762228.926911:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762228.926916:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762228.926921:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504168320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762228.926929:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762225/real 1789762228] req@ffff9018c8b9b800 x1876699504167808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762228.926944:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504167808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762228.926952:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762228.926956:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762228.926960:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504167808:192.168.203.158@tcp:250:kworker.0 00010000:00010000:3.0:1789762229.566206:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762230.590193:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762230.974164:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762230.974175:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762230.974180:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762230.974192:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2157 00000100:00080000:0.0:1789762230.974206:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762230.974215:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762230.974263:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504168576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762230.974291:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762230.974295:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762230.974297:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762230.974302:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2157 00000100:00080000:0.0:1789762230.974317:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762230.974323:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762230.974335:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762230.974342:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504168576:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762230.974354:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504168704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762230.974366:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762230.974371:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762230.974372:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762230.974376:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2157 00000100:00080000:0.0:1789762230.974380:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762230.974384:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762230.974389:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762230.974403:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504168832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762230.974413:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762230.974417:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762230.974419:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762230.974423:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2157 00000100:00080000:0.0:1789762230.974428:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762230.974434:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762230.974435:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762230.974442:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762230.974457:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504168960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762230.974466:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762230.974470:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762230.974472:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762230.974476:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2157 00000100:00080000:0.0:1789762230.974480:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762230.974485:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762230.974491:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762230.974502:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762230/real 1789762230] req@ffff9018c8b99f80 x1876699504168576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762230.974507:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504169088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762230.974536:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762230.974540:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762230.974543:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762230.974548:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504168576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762230.974551:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2157 00000100:00080000:0.0:1789762230.974558:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762230.974567:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789762230.974573:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789762230.974574:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762230.974583:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762230.974592:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504169216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762230.974593:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504168576:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762230.974634:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504168704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762230.974663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762230.974678:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762230/real 1789762230] req@ffff9018c8b98a80 x1876699504168704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762230.974700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504168704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762230.974710:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762230.974714:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762230.974717:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504168704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762230.974724:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504168832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762230.974748:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762230.974760:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762230/real 1789762230] req@ffff9018c8b99880 x1876699504168832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762230.974773:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504168832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762230.974782:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762230.974785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762230.974789:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504168832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762230.974796:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504168960:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762230.974820:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762230.974852:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762230/real 1789762230] req@ffff9018c8b9bb80 x1876699504168960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762230.974865:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504168960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762230.974874:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762230.974878:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762230.974882:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504168960:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762230.974888:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504169088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762230.974911:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762230.974923:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762230/real 1789762230] req@ffff9018c8b98700 x1876699504169088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762230.974934:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504169088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762230.974948:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762230.974951:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762230.974954:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504169088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762230.974960:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504169216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762230.974982:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762230.974992:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762230/real 1789762230] req@ffff9018c8b9b800 x1876699504169216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762230.975005:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504169216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762230.975012:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762230.975016:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762230.975019:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504169216:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762231.614261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762232.638211:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762233.662236:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762234.686261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762235.710371:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762236.094164:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762236.094175:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762236.094179:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762236.094189:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2162 00000100:00080000:3.0:1789762236.094215:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762236.094224:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762236.094274:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504169344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762236.094301:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762236.094305:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762236.094307:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762236.094312:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2162 00000100:00080000:3.0:1789762236.094317:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762236.094322:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762236.094335:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762236.094349:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504169472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762236.094358:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504169344:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762236.094360:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762236.094368:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762236.094371:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762236.094375:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2162 00000100:00080000:3.0:1789762236.094377:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762236.094380:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762236.094383:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762236.094392:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504169600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762236.094399:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762236.094402:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762236.094403:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762236.094417:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2162 00000100:00080000:3.0:1789762236.094419:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762236.094421:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762236.094425:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762236.094433:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504169728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762236.094440:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762236.094442:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762236.094444:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762236.094446:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2162 00000100:00080000:3.0:1789762236.094449:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762236.094452:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762236.094454:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762236.094464:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504169856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762236.094465:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762236.094474:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762236.094477:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762236.094478:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762236.094481:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2162 00000100:00080000:3.0:1789762236.094483:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762236.094485:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762236.094488:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762236.094495:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504169984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762236.094539:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762236/real 1789762236] req@ffff9018d070ce00 x1876699504169344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762236.094562:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504169344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762236.094574:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762236.094579:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762236.094584:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504169344:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762236.094641:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504169472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762236.094668:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762236.094694:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762236/real 1789762236] req@ffff9018d070ca80 x1876699504169472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762236.094708:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504169472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762236.094719:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762236.094723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762236.094727:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504169472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762236.094734:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504169600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762236.094767:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762236.094780:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762236/real 1789762236] req@ffff9018d070fb80 x1876699504169600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762236.094810:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504169600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762236.094820:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762236.094823:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762236.094827:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504169600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762236.094834:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504169728:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762236.094857:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762236.094869:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762236/real 1789762236] req@ffff9018d070f480 x1876699504169728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762236.094882:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504169728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762236.094892:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762236.094895:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762236.094899:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504169728:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762236.094906:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504169856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762236.094928:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762236.094939:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762236/real 1789762236] req@ffff9018d070ed80 x1876699504169856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762236.094961:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504169856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762236.094969:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762236.094972:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762236.094976:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504169856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762236.094982:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504169984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762236.095001:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762236.095011:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762236/real 1789762236] req@ffff9018d070df80 x1876699504169984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762236.095023:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504169984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762236.095032:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762236.095064:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762236.095068:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504169984:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762236.734199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762237.758298:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762238.782142:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762239.806167:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762240.830270:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762241.214242:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762241.214258:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762241.214265:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762241.214280:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2167 00000100:00080000:3.0:1789762241.214297:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762241.214308:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762241.214377:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504170112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762241.214428:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762241.214433:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762241.214435:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762241.214441:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2167 00000100:00080000:3.0:1789762241.214448:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762241.214455:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762241.214469:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762241.214486:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504170112:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762241.214491:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504170240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762241.214509:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762241.214514:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762241.214517:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762241.214522:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2167 00000100:00080000:3.0:1789762241.214527:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762241.214533:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762241.214550:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762241.214573:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504170368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762241.214589:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762241.214597:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762241.214598:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762241.214605:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762241.214612:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2167 00000100:00080000:3.0:1789762241.214619:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762241.214625:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762241.214649:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762241.214689:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504170496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762241.214725:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762241/real 1789762241] req@ffff9018d070d880 x1876699504170112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762241.214731:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762241.214779:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762241.214814:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762241.214822:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2167 00000100:00100000:2.0:1789762241.214829:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504170112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762241.214832:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762241.214845:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762241.214888:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762241.214890:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762241.214905:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762241.214912:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504170112:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762241.214922:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504170624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762241.214951:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762241.214959:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762241.214962:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762241.214972:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2167 00000100:00100000:2.0:1789762241.214975:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504170240:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762241.214992:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762241.215020:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762241.215029:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762241.215041:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762241.215051:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504170752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762241.215059:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762241/real 1789762241] req@ffff9018d070d180 x1876699504170240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762241.215143:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504170240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762241.215168:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762241.215175:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762241.215180:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504170240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762241.215190:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504170368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762241.215219:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762241.215233:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762241/real 1789762241] req@ffff9018d070e680 x1876699504170368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762241.215277:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504170368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762241.215287:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762241.215291:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762241.215294:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504170368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762241.215301:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504170496:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762241.215322:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762241.215334:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762241/real 1789762241] req@ffff9018d070df80 x1876699504170496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762241.215347:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504170496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762241.215383:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762241.215386:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762241.215388:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504170496:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762241.215393:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504170624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762241.215407:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762241.215415:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762241/real 1789762241] req@ffff9018d070ed80 x1876699504170624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762241.215425:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504170624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762241.215437:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762241.215440:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762241.215443:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504170624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762241.215458:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504170752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762241.215472:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762241.215480:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762241/real 1789762241] req@ffff9018d070f480 x1876699504170752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762241.215489:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504170752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762241.215496:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762241.215498:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762241.215501:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504170752:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762241.854184:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762242.878299:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762243.902211:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762244.926330:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762245.950168:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762246.334225:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762246.334233:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762246.334238:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762246.334249:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2172 00000100:00080000:0.0:1789762246.334261:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762246.334268:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762246.334309:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504170880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762246.334334:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762246.334337:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762246.334339:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762246.334342:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2172 00000100:00080000:0.0:1789762246.334345:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762246.334348:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762246.334358:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762246.334370:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504171008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762246.334377:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504170880:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762246.334383:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762246.334387:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762246.334390:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762246.334398:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2172 00000100:00080000:0.0:1789762246.334403:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762246.334406:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762246.334410:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762246.334419:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504171136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762246.334428:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762246.334431:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762246.334433:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762246.334436:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2172 00000100:00080000:0.0:1789762246.334440:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762246.334443:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762246.334448:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762246.334459:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504171264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762246.334459:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762246.334469:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762246.334472:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762246.334474:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762246.334477:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2172 00000100:00080000:0.0:1789762246.334479:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762246.334482:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762246.334484:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762246.334490:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504171392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762246.334496:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762246.334498:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762246.334499:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762246.334502:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2172 00000100:00080000:0.0:1789762246.334504:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762246.334506:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762246.334508:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762246.334513:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504171520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762246.334693:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762246/real 1789762246] req@ffff9018c8b9b100 x1876699504170880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762246.334725:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504170880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762246.334735:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762246.334739:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762246.334742:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504170880:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762246.334768:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504171008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762246.334795:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762246.334807:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762246/real 1789762246] req@ffff9018c8b9b480 x1876699504171008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762246.334815:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504171008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762246.334884:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762246.334887:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762246.334889:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504171008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762246.334894:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504171136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762246.334911:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762246.334919:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762246/real 1789762246] req@ffff9018c8b98000 x1876699504171136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762246.334927:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504171136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762246.334937:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762246.334939:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762246.334941:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504171136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762246.334945:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504171264:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762246.334956:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762246.334962:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762246/real 1789762246] req@ffff9018c8b9aa00 x1876699504171264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762246.334969:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504171264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762246.334975:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762246.334977:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762246.334979:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504171264:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762246.334982:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504171392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762246.334992:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762246.334998:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762246/real 1789762246] req@ffff9018c8b98e00 x1876699504171392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762246.335005:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504171392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762246.335010:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762246.335012:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762246.335014:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504171392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762246.335018:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504171520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762246.335028:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762246.335053:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762246/real 1789762246] req@ffff9018c8b9a680 x1876699504171520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762246.335063:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504171520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762246.335069:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762246.335072:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762246.335075:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504171520:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762246.974222:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762247.998160:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762249.022242:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762250.046200:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762251.070210:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762251.454152:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762251.454159:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762251.454163:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762251.454171:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2178 00000100:00080000:3.0:1789762251.454180:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762251.454186:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762251.454231:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504171648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762251.454253:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762251.454257:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762251.454258:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762251.454262:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2178 00000100:00080000:3.0:1789762251.454264:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762251.454268:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762251.454276:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762251.454285:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504171776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762251.454290:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504171648:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762251.454298:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762251.454302:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762251.454303:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762251.454306:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2178 00000100:00080000:3.0:1789762251.454308:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762251.454310:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762251.454313:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762251.454321:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504171904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762251.454326:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762251.454329:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762251.454330:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762251.454333:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2178 00000100:00080000:3.0:1789762251.454348:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762251.454350:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762251.454353:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789762251.454360:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762251.454362:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504172032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762251.454371:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762251.454373:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762251.454374:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762251.454377:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2178 00000100:00080000:3.0:1789762251.454379:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762251.454382:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762251.454385:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762251.454391:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504172160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762251.454398:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762251.454401:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762251.454402:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762251.454404:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2178 00000100:00080000:3.0:1789762251.454406:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762251.454409:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762251/real 1789762251] req@ffff9018d070fb80 x1876699504171648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762251.454411:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762251.454417:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762251.454427:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504172288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762251.454435:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504171648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762251.454451:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762251.454454:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762251.454457:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504171648:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762251.454477:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504171776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762251.454496:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762251.454505:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762251/real 1789762251] req@ffff9018d070ca80 x1876699504171776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762251.454513:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504171776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762251.454518:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762251.454521:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762251.454523:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504171776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762251.454526:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504171904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762251.454537:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762251.454542:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762251/real 1789762251] req@ffff9018d070ce00 x1876699504171904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762251.454549:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504171904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762251.454555:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762251.454556:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762251.454558:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504171904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762251.454562:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504172032:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762251.454572:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762251.454577:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762251/real 1789762251] req@ffff9018d070e300 x1876699504172032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762251.454584:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504172032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762251.454590:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762251.454591:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762251.454593:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504172032:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762251.454596:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504172160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762251.454605:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762251.454611:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762251/real 1789762251] req@ffff9018d070c000 x1876699504172160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762251.454618:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504172160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762251.454623:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762251.454625:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762251.454627:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504172160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762251.454630:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504172288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762251.454639:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762251.454644:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762251/real 1789762251] req@ffff9018d070dc00 x1876699504172288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762251.454652:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504172288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762251.454657:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762251.454658:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762251.454660:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504172288:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762252.094202:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762253.118288:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762254.142201:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762255.166288:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762256.190172:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762256.574203:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762256.574214:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762256.574218:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762256.574228:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2183 00000100:00080000:2.0:1789762256.574241:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762256.574248:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762256.574294:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504172416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762256.574333:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762256.574336:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762256.574337:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762256.574341:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2183 00000100:00080000:2.0:1789762256.574343:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762256.574347:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762256.574357:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762256.574367:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504172544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762256.574374:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504172416:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762256.574380:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762256.574384:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762256.574386:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762256.574389:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2183 00000100:00080000:2.0:1789762256.574391:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762256.574393:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762256.574397:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762256.574406:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504172672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762256.574412:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762256.574415:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762256.574416:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762256.574419:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2183 00000100:00080000:2.0:1789762256.574450:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762256.574454:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762256.574458:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789762256.574462:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762256.574468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504172800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762256.574476:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762256.574479:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762256.574481:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762256.574484:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2183 00000100:00080000:2.0:1789762256.574488:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762256.574491:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762256.574494:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762256.574503:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504172928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762256.574509:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762256.574514:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789762256.574514:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762256/real 1789762256] req@ffff9018d0f24380 x1876699504172416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762256.574518:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762256.574525:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2183 00000100:00080000:2.0:1789762256.574531:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762256.574533:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762256.574537:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762256.574541:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504172416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762256.574546:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504173056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762256.574565:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762256.574569:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762256.574573:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504172416:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762256.574602:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504172544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762256.574629:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762256.574655:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762256/real 1789762256] req@ffff9018d0f27800 x1876699504172544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762256.574665:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504172544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762256.574672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762256.574674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762256.574676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504172544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762256.574680:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504172672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762256.574695:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762256.574702:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762256/real 1789762256] req@ffff9018d0f25180 x1876699504172672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762256.574710:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504172672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762256.574720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762256.574722:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762256.574725:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504172672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762256.574728:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504172800:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762256.574739:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762256.574745:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762256/real 1789762256] req@ffff9018d0f24a80 x1876699504172800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762256.574751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504172800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762256.574757:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762256.574758:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762256.574760:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504172800:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762256.574764:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504172928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762256.574775:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762256.574781:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762256/real 1789762256] req@ffff9018d0f25500 x1876699504172928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762256.574789:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504172928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762256.574793:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762256.574795:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762256.574798:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504172928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762256.574802:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504173056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762256.574811:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762256.574817:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762256/real 1789762256] req@ffff9018d0f27480 x1876699504173056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762256.574824:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504173056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762256.574829:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762256.574831:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762256.574833:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504173056:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762257.214234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762258.238263:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762259.262277:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762260.286270:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762261.310241:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762261.694194:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762261.694204:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762261.694209:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762261.694220:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2188 00000100:00080000:3.0:1789762261.694232:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762261.694242:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762261.694291:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504173184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762261.694327:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762261.694331:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762261.694334:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762261.694339:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2188 00000100:00080000:3.0:1789762261.694346:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762261.694351:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762261.694362:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762261.694372:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504173184:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762261.694381:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504173312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762261.694394:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762261.694397:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762261.694399:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762261.694403:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2188 00000100:00080000:3.0:1789762261.694406:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762261.694410:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762261.694415:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762261.694427:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504173440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762261.694436:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762261.694440:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762261.694442:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762261.694449:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2188 00000800:00000100:2.0:1789762261.694450:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762261.694454:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762261.694460:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762261.694465:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762261.694480:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504173568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762261.694489:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762261.694493:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789762261.694496:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762261/real 1789762261] req@ffff9018d070d500 x1876699504173184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762261.694497:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762261.694509:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2188 00000100:00080000:3.0:1789762261.694518:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762261.694524:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762261.694529:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762261.694534:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504173184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762261.694545:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504173696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762261.694552:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762261.694564:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762261.694565:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762261.694574:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789762261.694576:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504173184:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762261.694577:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762261.694590:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2188 00000100:00080000:3.0:1789762261.694613:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762261.694618:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762261.694623:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762261.694647:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504173312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762261.694656:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504173824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762261.694684:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762261.694703:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762261/real 1789762261] req@ffff9018d070f800 x1876699504173312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762261.694719:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504173312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762261.694728:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762261.694731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762261.694735:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504173312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762261.694741:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504173440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762261.694785:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762261.694798:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762261/real 1789762261] req@ffff9018d070f100 x1876699504173440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762261.694810:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504173440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762261.694835:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762261.694838:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762261.694841:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504173440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762261.694847:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504173568:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762261.694870:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762261.694879:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762261/real 1789762261] req@ffff9018d070dc00 x1876699504173568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762261.694890:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504173568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762261.694897:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762261.694900:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762261.694903:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504173568:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762261.694907:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504173696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762261.694932:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762261.694951:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762261/real 1789762261] req@ffff9018d070c000 x1876699504173696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762261.694963:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504173696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762261.694970:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762261.694973:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762261.694976:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504173696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762261.694993:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504173824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762261.695009:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762261.695018:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762261/real 1789762261] req@ffff9018d070e300 x1876699504173824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762261.695062:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504173824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762261.695076:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762261.695079:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762261.695082:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504173824:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762262.334251:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762263.358206:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762264.382252:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762265.406198:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762266.430238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762266.814131:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762266.814140:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762266.814145:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762266.814154:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2193 00000100:00080000:3.0:1789762266.814166:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762266.814173:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762266.814208:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504173952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762266.814230:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762266.814233:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762266.814235:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762266.814239:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2193 00000100:00080000:3.0:1789762266.814243:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762266.814247:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762266.814256:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762266.814264:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504173952:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762266.814268:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504174080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762266.814279:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762266.814282:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762266.814283:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762266.814286:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2193 00000100:00080000:3.0:1789762266.814304:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762266.814307:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762266.814311:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762266.814321:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504174208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762266.814328:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762266.814331:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762266.814332:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762266.814335:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2193 00000100:00080000:3.0:1789762266.814337:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762266.814345:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762266.814350:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762266.814371:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762266.814382:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504174336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762266.814390:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762266.814392:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762266.814394:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762266.814397:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2193 00000100:00080000:3.0:1789762266.814401:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762266.814405:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762266.814409:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762266.814411:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762266/real 1789762266] req@ffff9018d070ce00 x1876699504173952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762266.814421:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504174464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762266.814431:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762266.814435:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504173952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762266.814436:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762266.814441:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762266.814450:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2193 00000100:00080000:3.0:1789762266.814456:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762266.814458:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762266.814460:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762266.814467:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762266.814468:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762266.814476:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504173952:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762266.814479:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504174592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762266.814505:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504174080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762266.814529:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762266.814540:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762266/real 1789762266] req@ffff9018d070ca80 x1876699504174080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762266.814549:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504174080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762266.814574:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762266.814579:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762266.814583:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504174080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762266.814615:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504174208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762266.814640:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762266.814652:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762266/real 1789762266] req@ffff9018d070fb80 x1876699504174208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762266.814661:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504174208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762266.814667:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762266.814670:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762266.814672:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504174208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762266.814676:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504174336:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762266.814690:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762266.814697:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762266/real 1789762266] req@ffff9018d070f480 x1876699504174336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762266.814705:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504174336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762266.814727:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762266.814729:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762266.814732:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504174336:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762266.814737:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504174464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762266.814750:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762266.814757:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762266/real 1789762266] req@ffff9018d070ed80 x1876699504174464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762266.814766:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504174464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762266.814772:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762266.814773:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762266.814776:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504174464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762266.814779:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504174592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762266.814790:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762266.814797:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762266/real 1789762266] req@ffff9018d070df80 x1876699504174592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762266.814805:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504174592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762266.814810:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762266.814812:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762266.814814:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504174592:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762267.454171:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762268.478252:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762269.502207:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762270.526204:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762271.550263:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762271.934095:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762271.934113:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762271.934117:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762271.934127:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2198 00000100:00080000:3.0:1789762271.934140:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762271.934146:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762271.934199:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504174720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762271.934222:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762271.934224:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762271.934226:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762271.934229:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2198 00000100:00080000:3.0:1789762271.934231:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762271.934234:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762271.934244:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762271.934254:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504174848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762271.934262:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762271.934265:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789762271.934266:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504174720:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762271.934266:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762271.934276:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2198 00000100:00080000:3.0:1789762271.934281:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762271.934284:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762271.934290:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762271.934302:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504174976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762271.934314:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762271.934318:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762271.934320:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762271.934325:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2198 00000100:00080000:3.0:1789762271.934338:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762271.934343:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789762271.934349:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789762271.934353:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762271.934373:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504175104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762271.934385:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762271.934389:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789762271.934392:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762271/real 1789762271] req@ffff9018d070e680 x1876699504174720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762271.934393:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762271.934403:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2198 00000100:00080000:3.0:1789762271.934410:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762271.934414:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762271.934418:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504174720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789762271.934420:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762271.934436:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504175232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762271.934439:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762271.934447:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762271.934448:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762271.934453:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762271.934457:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762271.934458:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504174720:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762271.934463:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2198 00000100:00080000:3.0:1789762271.934467:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762271.934470:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762271.934473:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762271.934480:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504175360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762271.934490:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504174848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762271.934513:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762271.934541:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762271/real 1789762271] req@ffff9018d070d180 x1876699504174848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762271.934556:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504174848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762271.934566:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762271.934570:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762271.934573:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504174848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762271.934578:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504174976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762271.934594:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762271.934602:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762271/real 1789762271] req@ffff9018d070d880 x1876699504174976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762271.934610:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504174976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762271.934616:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762271.934618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762271.934620:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504174976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762271.934625:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504175104:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762271.934636:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762271.934642:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762271/real 1789762271] req@ffff9018d070df80 x1876699504175104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762271.934650:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504175104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762271.934655:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762271.934657:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762271.934659:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504175104:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762271.934663:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504175232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762271.934673:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762271.934679:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762271/real 1789762271] req@ffff9018d070ed80 x1876699504175232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762271.934686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504175232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762271.934692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762271.934694:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762271.934695:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504175232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762271.934709:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504175360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762271.934719:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762271.934725:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762271/real 1789762271] req@ffff9018d070f480 x1876699504175360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762271.934733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504175360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762271.934739:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762271.934740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762271.934742:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504175360:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762272.574220:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762273.598199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762274.622227:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762275.646218:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762276.670278:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762277.054161:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762277.054171:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762277.054175:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762277.054186:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2203 00000100:00080000:2.0:1789762277.054199:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762277.054208:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762277.054262:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504175488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762277.054294:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762277.054297:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762277.054299:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762277.054302:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2203 00000100:00080000:2.0:1789762277.054305:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762277.054309:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762277.054320:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762277.054333:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504175616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762277.054347:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762277.054350:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762277.054351:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762277.054354:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2203 00000100:00080000:2.0:1789762277.054358:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762277.054362:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762277.054366:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504175488:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789762277.054367:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762277.054375:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504175744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762277.054388:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762277.054391:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762277.054392:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762277.054395:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2203 00000100:00080000:2.0:1789762277.054397:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762277.054399:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762277.054403:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762277.054413:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504175872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762277.054420:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762277.054422:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762277.054424:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762277.054426:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2203 00000100:00080000:2.0:1789762277.054444:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762277.054448:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762277.054451:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789762277.054459:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762277.054466:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504176000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762277.054475:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762277.054490:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762277.054491:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762277.054496:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2203 00000100:00080000:2.0:1789762277.054499:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762277.054502:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762277.054506:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762277.054518:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504176128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762277.054528:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762277/real 1789762277] req@ffff9018d0f26680 x1876699504175488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762277.054553:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504175488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762277.054567:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762277.054572:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762277.054577:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504175488:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762277.054603:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504175616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762277.054630:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762277.054644:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762277/real 1789762277] req@ffff9018d0f24700 x1876699504175616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762277.054659:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504175616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762277.054668:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762277.054673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762277.054676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504175616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762277.054684:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504175744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762277.054703:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762277.054713:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762277/real 1789762277] req@ffff9018d0f24000 x1876699504175744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762277.054727:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504175744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762277.054736:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762277.054740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762277.054744:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504175744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762277.054752:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504175872:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762277.054786:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762277.054798:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762277/real 1789762277] req@ffff9018d0f24e00 x1876699504175872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762277.054812:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504175872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762277.054822:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762277.054826:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762277.054830:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504175872:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762277.054837:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504176000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762277.054856:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762277.054868:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762277/real 1789762277] req@ffff9018d0f27480 x1876699504176000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762277.054881:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504176000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762277.054890:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762277.054894:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762277.054898:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504176000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762277.054905:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504176128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762277.054925:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762277.054935:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762277/real 1789762277] req@ffff9018d0f25500 x1876699504176128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762277.054949:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504176128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762277.054958:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762277.054962:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762277.054965:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504176128:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762277.694230:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762278.718184:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762279.742265:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762280.766188:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762281.790292:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762282.174168:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762282.174180:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762282.174184:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762282.174195:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2208 00000100:00080000:1.0:1789762282.174208:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762282.174216:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762282.174272:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504176256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762282.174308:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762282.174312:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762282.174313:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762282.174317:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2208 00000100:00080000:1.0:1789762282.174321:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762282.174324:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762282.174335:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762282.174349:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504176384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762282.174359:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762282.174363:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789762282.174363:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504176256:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762282.174366:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762282.174374:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2208 00000100:00080000:1.0:1789762282.174377:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762282.174380:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762282.174384:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762282.174401:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504176512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762282.174412:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762282.174416:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762282.174417:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762282.174421:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2208 00000100:00080000:1.0:1789762282.174427:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762282.174431:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762282.174435:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762282.174458:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504176640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762282.174465:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762282.174474:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762282.174478:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762282.174479:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762282.174484:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2208 00000100:00080000:1.0:1789762282.174489:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762282.174510:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762282.174515:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762282.174526:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504176768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762282.174534:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762282.174538:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762282.174539:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762282.174544:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2208 00000100:00080000:1.0:1789762282.174549:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762282.174552:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762282.174559:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762282.174572:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504176896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762282.174658:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762282/real 1789762282] req@ffff9018c66a5c00 x1876699504176256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762282.174850:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504176256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762282.174861:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762282.174865:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762282.174874:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504176256:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762282.174900:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504176384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762282.174983:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762282.175017:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762282/real 1789762282] req@ffff9018c66a6d80 x1876699504176384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762282.175044:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504176384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762282.175091:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762282.175095:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762282.175099:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504176384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762282.175106:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504176512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762282.175132:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762282.175144:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762282/real 1789762282] req@ffff9018c66a5f80 x1876699504176512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762282.175155:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504176512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762282.175170:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762282.175173:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762282.175176:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504176512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762282.175181:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504176640:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762282.175197:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762282.175206:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762282/real 1789762282] req@ffff9018c66a4000 x1876699504176640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762282.175215:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504176640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762282.175227:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762282.175230:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762282.175232:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504176640:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762282.175236:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504176768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762282.175249:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762282.175257:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762282/real 1789762282] req@ffff9018c66a7b80 x1876699504176768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762282.175281:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504176768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762282.175286:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762282.175288:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762282.175290:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504176768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762282.175293:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504176896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762282.175305:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762282.175311:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762282/real 1789762282] req@ffff9018c66a7800 x1876699504176896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762282.175318:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504176896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762282.175323:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762282.175324:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762282.175327:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504176896:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762282.814164:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762283.838268:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762284.862255:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762285.886339:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762286.910192:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762287.294196:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762287.294218:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762287.294222:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762287.294233:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2213 00000100:00080000:1.0:1789762287.294245:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762287.294253:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762287.294296:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504177024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762287.294330:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762287.294334:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762287.294336:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762287.294341:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2213 00000100:00080000:1.0:1789762287.294344:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762287.294347:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762287.294358:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762287.294371:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504177152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762287.294382:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762287.294387:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762287.294389:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762287.294393:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2213 00000100:00080000:1.0:1789762287.294396:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762287.294415:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762287.294422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762287.294435:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504177024:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762287.294438:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504177280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762287.294459:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762287.294462:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762287.294464:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762287.294469:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2213 00000100:00080000:1.0:1789762287.294474:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762287.294478:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762287.294484:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762287.294498:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504177408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762287.294508:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762287.294513:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762287.294515:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789762287.294520:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762287.294532:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2213 00000100:00080000:1.0:1789762287.294536:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762287.294542:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762287.294548:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762287.294561:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504177536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762287.294573:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789762287.294580:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762287/real 1789762287] req@ffff9018c66a5500 x1876699504177024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762287.294581:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762287.294586:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762287.294598:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2213 00000100:00080000:1.0:1789762287.294616:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789762287.294622:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504177024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762287.294624:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762287.294635:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789762287.294649:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789762287.294657:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504177664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762287.294662:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762287.294674:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504177024:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762287.294690:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504177152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762287.294719:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762287.294732:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762287/real 1789762287] req@ffff9018c66a4e00 x1876699504177152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762287.294745:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504177152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762287.294753:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762287.294757:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762287.294760:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504177152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762287.294790:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504177280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762287.294807:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762287.294817:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762287/real 1789762287] req@ffff9018c66a6300 x1876699504177280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762287.294829:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504177280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762287.294837:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762287.294839:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762287.294843:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504177280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762287.294848:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504177408:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762287.294863:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762287.294872:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762287/real 1789762287] req@ffff9018c66a4700 x1876699504177408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762287.294883:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504177408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762287.294891:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762287.294894:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762287.294897:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504177408:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762287.294902:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504177536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762287.294916:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762287.294925:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762287/real 1789762287] req@ffff9018c66a6680 x1876699504177536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762287.294937:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504177536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762287.294945:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762287.294947:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762287.294950:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504177536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762287.294955:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504177664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762287.294969:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762287.294978:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762287/real 1789762287] req@ffff9018c66a5180 x1876699504177664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762287.294993:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504177664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762287.295007:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762287.295012:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762287.295017:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504177664:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762287.934240:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762288.958183:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762289.982212:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762291.006222:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762292.030278:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762292.414126:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762292.414136:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762292.414141:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762292.414151:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2218 00000100:00080000:1.0:1789762292.414164:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762292.414171:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762292.414227:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504177792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762292.414268:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762292.414271:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762292.414272:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762292.414275:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2218 00000100:00080000:1.0:1789762292.414278:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762292.414282:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762292.414292:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762292.414319:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504177920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762292.414331:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762292.414335:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762292.414337:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762292.414341:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504177792:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762292.414345:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2218 00000100:00080000:1.0:1789762292.414352:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762292.414357:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762292.414373:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762292.414388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504178048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762292.414400:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762292.414404:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762292.414405:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762292.414419:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2218 00000100:00080000:1.0:1789762292.414422:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762292.414426:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762292.414431:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762292.414440:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504178176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762292.414452:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762292.414454:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762292.414456:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762292.414458:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2218 00000100:00080000:1.0:1789762292.414461:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762292.414464:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762292.414468:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762292.414478:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504178304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762292.414484:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504177920:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789762292.414488:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762292.414492:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762292.414494:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762292.414498:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2218 00000100:00080000:1.0:1789762292.414500:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762292.414503:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762292.414507:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762292.414516:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504178048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762292.414520:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504178432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762292.414545:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504178176:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762292.414565:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504178304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762292.414581:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504178432:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762293.054146:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762294.078266:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762295.102162:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762295.486332:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789762295.486403:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762295.486423:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762295.486467:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762295.486480:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762295.486489:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762295.486500:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789762295.486507:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762292/real 1789762295] req@ffff9018c66a7100 x1876699504177920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1789762295.486508:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789762295.486537:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504177920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762295.486548:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762295.486551:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762295.486555:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504177920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762295.486561:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762292/real 1789762295] req@ffff9018c66a7480 x1876699504178048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762295.486571:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504178048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762295.486576:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762295.486579:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762295.486581:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504178048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762295.486586:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762292/real 1789762295] req@ffff9018c66a4a80 x1876699504178176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762295.486603:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504178176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762295.486614:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762295.486618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762295.486623:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504178176:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762295.486631:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762292/real 1789762295] req@ffff9018c66a4380 x1876699504177792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762295.486667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504177792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762295.486677:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762295.486681:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762295.486686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504177792:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762295.486715:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762292/real 1789762295] req@ffff9018c66a6a00 x1876699504178304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762295.486733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504178304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762295.486744:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762295.486748:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762295.486752:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504178304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762295.486762:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762292/real 1789762295] req@ffff9018c66a5880 x1876699504178432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762295.486776:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504178432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762295.486793:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762295.486797:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762295.486801:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504178432:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762296.126245:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762297.150212:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762297.534159:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762297.534170:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762297.534175:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762297.534187:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2224 00000100:00080000:1.0:1789762297.534199:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762297.534207:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762297.534250:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504178560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762297.534279:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762297.534282:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762297.534284:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762297.534289:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2224 00000100:00080000:1.0:1789762297.534293:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762297.534298:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762297.534311:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762297.534338:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504178688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762297.534344:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504178560:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762297.534359:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762297.534364:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762297.534366:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762297.534371:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2224 00000100:00080000:1.0:1789762297.534376:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762297.534380:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762297.534386:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762297.534402:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504178816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762297.534413:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762297.534417:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762297.534419:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762297.534423:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2224 00000100:00080000:1.0:1789762297.534427:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762297.534432:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762297.534438:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789762297.534443:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762297.534454:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504178944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762297.534467:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762297.534471:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762297.534473:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762297.534477:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2224 00000100:00080000:1.0:1789762297.534482:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762297.534486:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762297.534492:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762297.534502:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762297/real 1789762297] req@ffff9018c66a5880 x1876699504178560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762297.534508:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504179072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762297.534532:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762297.534536:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762297.534538:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762297.534542:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504178560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762297.534547:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2224 00000100:00080000:1.0:1789762297.534554:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762297.534561:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789762297.534562:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1789762297.534568:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789762297.534571:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762297.534578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504178560:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762297.534586:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504179200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762297.534614:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504178688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762297.534643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762297.534660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762297/real 1789762297] req@ffff9018c66a6a00 x1876699504178688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762297.534674:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504178688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762297.534683:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762297.534687:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762297.534690:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504178688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762297.534697:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504178816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762297.534718:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762297.534730:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762297/real 1789762297] req@ffff9018c66a4380 x1876699504178816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762297.534742:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504178816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762297.534751:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762297.534754:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762297.534757:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504178816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762297.534762:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504178944:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762297.534784:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762297.534796:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762297/real 1789762297] req@ffff9018c66a4a80 x1876699504178944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762297.534806:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504178944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762297.534814:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762297.534817:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762297.534821:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504178944:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762297.534827:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504179072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762297.534845:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762297.534856:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762297/real 1789762297] req@ffff9018c66a7480 x1876699504179072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762297.534878:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504179072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762297.534886:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762297.534889:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762297.534894:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504179072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762297.534899:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504179200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762297.534919:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762297.534931:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762297/real 1789762297] req@ffff9018c66a7100 x1876699504179200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762297.534941:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504179200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762297.534947:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762297.534950:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762297.534952:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504179200:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762298.174204:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762299.198289:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762300.222199:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762301.246233:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762302.270200:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762302.654098:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762302.654122:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762302.654126:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762302.654136:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2229 00000100:00080000:1.0:1789762302.654148:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762302.654155:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762302.654214:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504179328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762302.654241:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762302.654244:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762302.654246:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762302.654249:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2229 00000100:00080000:1.0:1789762302.654251:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762302.654253:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762302.654263:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762302.654272:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504179456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762302.654280:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762302.654282:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762302.654284:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762302.654312:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2229 00000100:00080000:1.0:1789762302.654316:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762302.654319:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762302.654323:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762302.654329:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504179328:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762302.654336:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504179584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762302.654344:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762302.654347:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762302.654348:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762302.654351:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2229 00000100:00080000:1.0:1789762302.654353:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762302.654356:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762302.654359:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762302.654377:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504179712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762302.654384:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762302.654387:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762302.654388:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762302.654390:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2229 00000100:00080000:1.0:1789762302.654395:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762302.654397:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762302.654400:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762302.654407:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504179840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762302.654418:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762302.654421:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762302.654422:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762302.654427:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2229 00000800:00000100:3.0:1789762302.654428:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762302.654430:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762302.654434:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762302.654437:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762302.654443:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504179968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762302.654493:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762302/real 1789762302] req@ffff9018c66a5180 x1876699504179328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762302.654518:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504179328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762302.654531:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762302.654535:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762302.654540:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504179328:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762302.654549:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504179456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762302.654576:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762302.654591:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762302/real 1789762302] req@ffff9018c66a6680 x1876699504179456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762302.654605:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504179456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762302.654614:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762302.654618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762302.654621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504179456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762302.654654:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504179584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762302.654688:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762302.654701:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762302/real 1789762302] req@ffff9018c66a4700 x1876699504179584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762302.654716:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504179584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762302.654725:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762302.654728:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762302.654732:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504179584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762302.654738:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504179712:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762302.654761:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762302.654774:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762302/real 1789762302] req@ffff9018c66a6300 x1876699504179712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762302.654787:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504179712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762302.654796:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762302.654800:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762302.654804:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504179712:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762302.654810:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504179840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762302.654831:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762302.654844:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762302/real 1789762302] req@ffff9018c66a4e00 x1876699504179840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762302.654856:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504179840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762302.654871:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762302.654875:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762302.654879:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504179840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762302.654886:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504179968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762302.654917:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762302.654928:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762302/real 1789762302] req@ffff9018c66a5500 x1876699504179968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762302.654941:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504179968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762302.654949:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762302.654953:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762302.654957:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504179968:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762303.294230:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762304.318229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762305.342232:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762306.366328:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762307.390208:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762307.774128:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762307.774137:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762307.774141:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762307.774149:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2234 00000100:00080000:0.0:1789762307.774158:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762307.774175:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762307.774212:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504180096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762307.774243:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762307.774265:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762307.774267:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762307.774272:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2234 00000100:00080000:0.0:1789762307.774276:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762307.774279:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762307.774299:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762307.774315:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504180224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762307.774324:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762307.774327:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762307.774328:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762307.774333:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504180096:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762307.774335:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2234 00000100:00080000:0.0:1789762307.774341:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762307.774346:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762307.774348:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762307.774367:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504180352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762307.774374:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762307.774377:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762307.774378:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762307.774381:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2234 00000100:00080000:0.0:1789762307.774383:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762307.774385:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762307.774389:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762307.774395:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504180480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762307.774401:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762307.774405:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762307.774406:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762307.774408:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2234 00000100:00080000:0.0:1789762307.774411:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762307.774416:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789762307.774417:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789762307.774420:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762307.774428:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504180608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762307.774440:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762307.774442:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762307.774444:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762307.774447:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2234 00000100:00080000:0.0:1789762307.774449:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762307.774452:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762307.774455:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762307.774458:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762307/real 1789762307] req@ffff9018c8b9a300 x1876699504180096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762307.774466:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504180736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762307.774486:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504180096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762307.774501:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762307.774504:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762307.774507:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504180096:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762307.774527:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504180224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762307.774543:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762307.774552:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762307/real 1789762307] req@ffff9018c8b9ad80 x1876699504180224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762307.774561:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504180224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762307.774567:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762307.774569:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762307.774572:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504180224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762307.774575:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504180352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762307.774585:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762307.774590:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762307/real 1789762307] req@ffff9018c8b99500 x1876699504180352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762307.774608:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504180352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762307.774614:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762307.774615:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762307.774617:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504180352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762307.774622:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504180480:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762307.774641:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762307.774653:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762307/real 1789762307] req@ffff9018c8b99c00 x1876699504180480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762307.774664:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504180480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762307.774683:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762307.774686:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762307.774689:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504180480:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762307.774695:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504180608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762307.774715:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762307.774726:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762307/real 1789762307] req@ffff9018c8b98380 x1876699504180608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762307.774736:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504180608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762307.774743:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762307.774746:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762307.774749:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504180608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762307.774755:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504180736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762307.774772:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762307.774781:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762307/real 1789762307] req@ffff9018c8b9a680 x1876699504180736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762307.774803:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504180736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762307.774812:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762307.774815:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762307.774825:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504180736:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762308.414168:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762309.438336:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762310.462362:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762311.486256:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762312.510158:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762312.894186:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762312.894199:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762312.894204:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762312.894247:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2239 00000100:00080000:1.0:1789762312.894271:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762312.894279:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762312.894339:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504180864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762312.894388:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762312.894393:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762312.894395:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762312.894400:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2239 00000100:00080000:1.0:1789762312.894405:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762312.894411:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762312.894425:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762312.894433:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504180864:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762312.894442:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504180992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762312.894454:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762312.894457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762312.894459:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762312.894462:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2239 00000100:00080000:1.0:1789762312.894466:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762312.894469:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762312.894474:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762312.894484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504181120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762312.894494:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762312.894506:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762312.894511:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762312.894514:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762312.894520:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2239 00000100:00080000:1.0:1789762312.894526:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762312.894531:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762312.894537:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762312/real 1789762312] req@ffff9018c66a7800 x1876699504180864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762312.894541:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762312.894563:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504181248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762312.894566:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504180864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762312.894583:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762312.894591:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762312.894593:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762312.894595:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762312.894600:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2239 00000100:00080000:1.0:1789762312.894605:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762312.894606:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789762312.894612:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762312.894619:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504180864:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789762312.894620:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762312.894660:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504181376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762312.894668:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504180992:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789762312.894689:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762312.894694:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762312.894698:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762312.894706:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2239 00000800:00000100:3.0:1789762312.894713:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762312.894717:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762312.894727:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762312.894734:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762312.894741:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762312/real 1789762312] req@ffff9018c66a7b80 x1876699504180992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762312.894751:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504181504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762312.894782:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504180992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762312.894793:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762312.894797:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762312.894801:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504180992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762312.894809:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504181120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762312.894832:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762312.894843:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762312/real 1789762312] req@ffff9018c66a4000 x1876699504181120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762312.894854:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504181120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762312.894862:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762312.894865:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762312.894868:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504181120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762312.894873:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504181248:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762312.894886:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762312.894894:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762312/real 1789762312] req@ffff9018c66a5f80 x1876699504181248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762312.894904:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504181248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762312.894918:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762312.894921:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762312.894923:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504181248:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762312.894955:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504181376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762312.894968:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762312.894976:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762312/real 1789762312] req@ffff9018c66a6d80 x1876699504181376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762312.894984:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504181376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762312.894991:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762312.894994:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762312.894996:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504181376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762312.895001:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504181504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762312.895011:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762312.895018:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762312/real 1789762312] req@ffff9018c66a5c00 x1876699504181504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762312.895026:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504181504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762312.895032:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762312.895035:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762312.895038:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504181504:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762313.534298:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762314.558158:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762315.582260:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762316.606201:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762317.630231:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762318.014132:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762318.014140:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762318.014145:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762318.014153:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2244 00000100:00080000:2.0:1789762318.014163:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762318.014185:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762318.014217:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504181632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762318.014242:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762318.014245:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762318.014247:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762318.014250:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2244 00000100:00080000:2.0:1789762318.014253:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762318.014256:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762318.014264:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762318.014276:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504181760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762318.014283:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504181632:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762318.014286:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762318.014293:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762318.014295:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762318.014300:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2244 00000100:00080000:2.0:1789762318.014303:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762318.014306:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762318.014309:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762318.014319:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504181888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762318.014330:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762318.014332:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762318.014333:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762318.014337:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2244 00000100:00080000:2.0:1789762318.014339:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762318.014341:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762318.014344:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762318.014353:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504182016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762318.014361:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762318.014362:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762318.014364:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762318.014367:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762318.014371:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2244 00000100:00080000:2.0:1789762318.014374:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762318.014377:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762318.014381:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762318.014388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504182144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762318.014394:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762318.014397:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762318.014399:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762318.014401:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2244 00000100:00080000:2.0:1789762318.014403:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762318.014406:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762318.014409:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762318.014416:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762318/real 1789762318] req@ffff9018d0f24a80 x1876699504181632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762318.014420:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504182272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762318.014453:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504181632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762318.014464:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762318.014469:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762318.014472:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504181632:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762318.014497:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504181760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762318.014521:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762318.014535:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762318/real 1789762318] req@ffff9018d0f25180 x1876699504181760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762318.014548:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504181760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762318.014558:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762318.014561:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762318.014565:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504181760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762318.014572:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504181888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762318.014590:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762318.014598:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762318/real 1789762318] req@ffff9018d0f27800 x1876699504181888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762318.014612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504181888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762318.014621:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762318.014625:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762318.014629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504181888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762318.014636:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504182016:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762318.014661:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762318.014674:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762318/real 1789762318] req@ffff9018d0f24380 x1876699504182016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762318.014686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504182016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762318.014693:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762318.014696:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762318.014698:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504182016:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762318.014704:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504182144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762318.014723:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762318.014737:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762318/real 1789762318] req@ffff9018d0f26a00 x1876699504182144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762318.014747:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504182144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762318.014754:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762318.014756:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762318.014759:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504182144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762318.014764:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504182272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762318.014777:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762318.014785:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762318/real 1789762318] req@ffff9018d0f25c00 x1876699504182272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762318.014797:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504182272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762318.014802:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762318.014804:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762318.014805:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504182272:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762318.654270:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762319.678154:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762320.702258:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762321.726194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762322.750240:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762323.134118:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762323.134126:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762323.134130:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762323.134150:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2249 00000100:00080000:2.0:1789762323.134159:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762323.134165:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762323.134201:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504182400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762323.134230:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762323.134233:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762323.134234:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762323.134237:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2249 00000100:00080000:2.0:1789762323.134240:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762323.134244:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762323.134252:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762323.134263:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504182528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762323.134271:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504182400:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762323.134274:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762323.134278:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762323.134279:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762323.134282:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2249 00000100:00080000:2.0:1789762323.134284:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762323.134286:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762323.134289:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762323.134297:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504182656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762323.134303:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762323.134306:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762323.134307:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762323.134310:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2249 00000100:00080000:2.0:1789762323.134313:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762323.134315:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762323.134319:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762323.134326:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504182784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762323.134332:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762323.134339:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762323.134342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762323.134343:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762323.134347:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2249 00000100:00080000:2.0:1789762323.134351:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762323.134354:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762323.134358:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762323.134369:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762323/real 1789762323] req@ffff9018d0f26300 x1876699504182400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762323.134373:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504182912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762323.134396:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762323.134397:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504182400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762323.134406:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762323.134411:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762323.134416:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762323.134419:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2249 00000100:00080000:1.0:1789762323.134423:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789762323.134427:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762323.134428:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504182400:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762323.134446:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762323.134452:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504182528:192.168.203.158@tcp:38:kworker.0 00000008:00100000:2.0:1789762323.134453:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762323.134468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504183040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762323.134481:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762323.134495:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762323/real 1789762323] req@ffff9018d0f27100 x1876699504182528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762323.134505:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504182528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762323.134516:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762323.134518:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762323.134521:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504182528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762323.134525:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504182656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762323.134536:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762323.134543:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762323/real 1789762323] req@ffff9018d0f25f80 x1876699504182656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762323.134550:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504182656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762323.134555:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762323.134557:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762323.134559:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504182656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762323.134563:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504182784:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762323.134573:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762323.134579:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762323/real 1789762323] req@ffff9018d0f26d80 x1876699504182784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762323.134587:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504182784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762323.134591:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762323.134594:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762323.134598:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504182784:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762323.134604:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504182912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762323.134624:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762323.134635:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762323/real 1789762323] req@ffff9018d0f25c00 x1876699504182912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762323.134646:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504182912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762323.134654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762323.134657:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762323.134660:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504182912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762323.134676:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504183040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762323.134694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762323.134706:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762323/real 1789762323] req@ffff9018d0f26a00 x1876699504183040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762323.134717:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504183040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762323.134724:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762323.134728:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762323.134730:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504183040:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762323.774224:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762324.798262:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762325.822198:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762326.846191:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762327.870184:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762328.254230:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762328.254243:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762328.254248:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762328.254258:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2254 00000100:00080000:3.0:1789762328.254269:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762328.254277:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762328.254330:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504183168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762328.254358:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762328.254361:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762328.254362:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762328.254366:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2254 00000100:00080000:3.0:1789762328.254368:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762328.254370:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762328.254380:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762328.254391:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504183296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762328.254402:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762328.254406:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762328.254408:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762328.254413:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2254 00000100:00080000:3.0:1789762328.254418:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762328.254423:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762328.254429:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762328.254433:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504183168:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762328.254442:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504183424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762328.254451:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762328.254454:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762328.254455:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762328.254458:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2254 00000100:00080000:3.0:1789762328.254461:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762328.254463:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762328.254468:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762328.254477:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504183552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762328.254484:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762328.254486:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762328.254488:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762328.254491:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2254 00000100:00080000:3.0:1789762328.254494:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762328.254497:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762328.254501:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762328.254507:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504183680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762328.254514:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762328.254517:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762328.254518:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762328.254520:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2254 00000100:00080000:3.0:1789762328.254529:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762328.254530:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762328.254535:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762328.254539:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762328.254547:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504183808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762328.254588:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762328/real 1789762328] req@ffff9018d070fb80 x1876699504183168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762328.254618:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504183168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762328.254631:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762328.254636:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762328.254640:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504183168:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762328.254656:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504183296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762328.254682:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762328.254696:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762328/real 1789762328] req@ffff9018d070ca80 x1876699504183296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762328.254710:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504183296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762328.254718:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762328.254722:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762328.254725:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504183296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762328.254764:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504183424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762328.254786:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762328.254799:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762328/real 1789762328] req@ffff9018d070ce00 x1876699504183424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762328.254812:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504183424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762328.254828:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762328.254831:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762328.254835:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504183424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762328.254841:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504183552:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762328.254864:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762328.254873:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762328/real 1789762328] req@ffff9018d070e300 x1876699504183552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762328.254880:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504183552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762328.254886:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762328.254888:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762328.254890:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504183552:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762328.254894:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504183680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762328.254906:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762328.254912:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762328/real 1789762328] req@ffff9018d070c000 x1876699504183680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762328.254920:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504183680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762328.254924:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762328.254926:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762328.254929:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504183680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762328.254932:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504183808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762328.254943:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762328.254948:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762328/real 1789762328] req@ffff9018d070dc00 x1876699504183808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762328.254965:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504183808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762328.254970:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762328.254972:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762328.254974:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504183808:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762328.894243:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762329.918178:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762330.942199:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762331.966160:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762332.990276:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762333.374163:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762333.374173:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762333.374178:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762333.374189:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2259 00000100:00080000:3.0:1789762333.374202:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762333.374211:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762333.374256:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504183936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762333.374295:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762333.374301:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762333.374303:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762333.374309:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2259 00000100:00080000:3.0:1789762333.374314:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762333.374320:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762333.374332:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762333.374358:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504184064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762333.374369:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762333.374372:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762333.374374:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762333.374377:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2259 00000100:00080000:3.0:1789762333.374381:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762333.374384:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762333.374389:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762333.374403:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504183936:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762333.374404:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504184192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762333.374443:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762333.374446:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762333.374448:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762333.374452:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2259 00000100:00080000:3.0:1789762333.374455:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762333.374458:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762333.374463:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762333.374487:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504184320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762333.374497:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762333.374500:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762333.374501:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762333.374506:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2259 00000100:00080000:3.0:1789762333.374508:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762333.374511:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762333.374515:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762333.374523:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504184448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762333.374531:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762333.374534:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762333.374535:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762333.374538:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2259 00000100:00080000:3.0:1789762333.374550:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762333.374554:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762333.374557:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762333.374569:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504184576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762333.374580:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762333.374666:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762333/real 1789762333] req@ffff9018d070f100 x1876699504183936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762333.374700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504183936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762333.374719:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762333.374726:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762333.374733:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504183936:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762333.374747:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504184064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762333.374786:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762333.374809:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762333/real 1789762333] req@ffff9018d070f800 x1876699504184064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762333.374829:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504184064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762333.374843:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762333.374848:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762333.374854:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504184064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762333.374895:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504184192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762333.374929:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762333.374948:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762333/real 1789762333] req@ffff9018d070d500 x1876699504184192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762333.374968:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504184192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762333.374982:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762333.374987:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762333.374992:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504184192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762333.375004:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504184320:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762333.375040:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762333.375124:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762333/real 1789762333] req@ffff9018d070dc00 x1876699504184320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762333.375147:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504184320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762333.375162:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762333.375168:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762333.375175:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504184320:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762333.375185:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504184448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762333.375222:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762333.375242:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762333/real 1789762333] req@ffff9018d070c000 x1876699504184448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762333.375262:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504184448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762333.375277:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762333.375282:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762333.375288:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504184448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762333.375300:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504184576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762333.375337:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762333.375357:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762333/real 1789762333] req@ffff9018d070e300 x1876699504184576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762333.375376:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504184576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762333.375390:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762333.375395:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762333.375401:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504184576:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762334.014202:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762335.038201:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762336.062231:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762337.086220:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762338.110256:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762338.494171:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762338.494182:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762338.494187:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762338.494197:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2265 00000100:00080000:0.0:1789762338.494208:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762338.494226:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762338.494276:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504184704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762338.494307:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762338.494310:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762338.494312:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762338.494315:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2265 00000100:00080000:0.0:1789762338.494318:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762338.494322:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762338.494332:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762338.494344:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504184832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762338.494352:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762338.494357:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789762338.494358:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504184704:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762338.494359:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762338.494366:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2265 00000100:00080000:0.0:1789762338.494369:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762338.494371:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762338.494374:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762338.494384:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504184960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762338.494390:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762338.494393:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762338.494394:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762338.494397:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2265 00000100:00080000:0.0:1789762338.494400:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762338.494402:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762338.494406:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762338.494413:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504185088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762338.494420:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762338.494423:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762338.494424:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762338.494426:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2265 00000100:00080000:0.0:1789762338.494429:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762338.494431:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762338.494434:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762338.494438:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762338.494445:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504185216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762338.494452:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762338.494454:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762338.494456:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762338.494459:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2265 00000100:00080000:0.0:1789762338.494462:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762338.494464:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762338.494467:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762338.494474:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504185344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762338.494490:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762338/real 1789762338] req@ffff9018c8b98e00 x1876699504184704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762338.494506:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504184704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762338.494515:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762338.494518:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762338.494521:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504184704:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762338.494543:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504184832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762338.494560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762338.494569:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762338/real 1789762338] req@ffff9018c8b9aa00 x1876699504184832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762338.494577:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504184832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762338.494582:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762338.494584:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762338.494587:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504184832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762338.494594:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504184960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762338.494615:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762338.494627:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762338/real 1789762338] req@ffff9018c8b98000 x1876699504184960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762338.494640:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504184960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762338.494650:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762338.494653:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762338.494656:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504184960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762338.494662:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504185088:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762338.494679:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762338.494687:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762338/real 1789762338] req@ffff9018c8b9b480 x1876699504185088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762338.494695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504185088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762338.494700:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762338.494702:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762338.494705:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504185088:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762338.494708:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504185216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762338.494718:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762338.494724:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762338/real 1789762338] req@ffff9018c8b9b100 x1876699504185216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762338.494731:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504185216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762338.494737:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762338.494739:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762338.494741:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504185216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762338.494744:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504185344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762338.494754:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762338.494759:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762338/real 1789762338] req@ffff9018c8b9b800 x1876699504185344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762338.494766:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504185344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762338.494770:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762338.494772:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762338.494774:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504185344:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762339.134184:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762340.158214:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762341.182272:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762342.206240:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762343.230224:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762343.614185:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762343.614199:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762343.614204:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762343.614218:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2270 00000100:00080000:1.0:1789762343.614232:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762343.614241:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762343.614296:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504185472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762343.614323:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762343.614338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762343.614340:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762343.614343:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2270 00000100:00080000:1.0:1789762343.614346:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762343.614349:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762343.614359:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762343.614370:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504185600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762343.614378:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762343.614381:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762343.614384:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762343.614385:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504185472:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762343.614388:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2270 00000100:00080000:1.0:1789762343.614405:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762343.614408:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762343.614412:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762343.614420:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504185728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762343.614426:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762343.614429:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762343.614431:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762343.614434:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2270 00000100:00080000:1.0:1789762343.614436:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762343.614439:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762343.614443:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762343.614451:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504185856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762343.614457:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762343.614460:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762343.614461:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762343.614463:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2270 00000100:00080000:1.0:1789762343.614466:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762343.614468:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762343.614472:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789762343.614472:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762343.614484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504185984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762343.614491:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762343.614493:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762343.614495:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762343.614498:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2270 00000100:00080000:1.0:1789762343.614501:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762343.614503:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762343.614516:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762343.614524:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504186112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762343.614535:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762343/real 1789762343] req@ffff9018c66a5c00 x1876699504185472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762343.614557:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504185472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762343.614581:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762343.614586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762343.614590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504185472:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762343.614622:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504185600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762343.614643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762343.614653:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762343/real 1789762343] req@ffff9018c66a6d80 x1876699504185600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762343.614666:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504185600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762343.614675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762343.614680:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762343.614684:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504185600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762343.614691:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504185728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762343.614713:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762343.614723:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762343/real 1789762343] req@ffff9018c66a5f80 x1876699504185728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762343.614732:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504185728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762343.614738:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762343.614740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762343.614742:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504185728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762343.614746:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504185856:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762343.614760:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762343.614766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762343/real 1789762343] req@ffff9018c66a4000 x1876699504185856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762343.614774:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504185856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762343.614780:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762343.614781:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762343.614783:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504185856:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762343.614787:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504185984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762343.614798:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762343.614804:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762343/real 1789762343] req@ffff9018c66a7b80 x1876699504185984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762343.614812:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504185984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762343.614817:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762343.614819:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762343.614820:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504185984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762343.614824:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504186112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762343.614834:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762343.614839:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762343/real 1789762343] req@ffff9018c66a7800 x1876699504186112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762343.614846:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504186112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762343.614851:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762343.614853:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762343.614855:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504186112:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762344.254229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762345.278298:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762346.302293:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762347.326289:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762348.350312:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762348.734152:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762348.734164:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762348.734170:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762348.734182:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2275 00000100:00080000:1.0:1789762348.734196:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762348.734205:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762348.734254:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504186240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762348.734306:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762348.734310:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762348.734312:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762348.734316:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2275 00000100:00080000:1.0:1789762348.734319:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762348.734323:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762348.734336:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762348.734350:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504186368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762348.734361:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762348.734368:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762348.734370:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762348.734370:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504186240:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762348.734381:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2275 00000100:00080000:1.0:1789762348.734386:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762348.734389:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762348.734393:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762348.734406:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504186496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762348.734421:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762348.734424:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762348.734426:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762348.734429:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2275 00000100:00080000:1.0:1789762348.734432:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762348.734436:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762348.734440:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762348.734451:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504186624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762348.734460:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762348.734464:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762348.734465:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762348.734468:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2275 00000100:00080000:1.0:1789762348.734472:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789762348.734478:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762348.734481:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762348.734488:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762348.734508:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504186752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762348.734518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762348.734521:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762348.734523:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762348.734527:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2275 00000100:00080000:1.0:1789762348.734531:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762348.734534:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762348.734539:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762348.734556:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504186880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762348.734562:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762348/real 1789762348] req@ffff9018c66a5500 x1876699504186240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762348.734614:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504186240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762348.734644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762348.734652:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762348.734659:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504186240:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762348.734711:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504186368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762348.734758:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762348.734775:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762348/real 1789762348] req@ffff9018c66a4e00 x1876699504186368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762348.734788:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504186368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762348.734797:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762348.734800:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762348.734803:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504186368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762348.734809:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504186496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762348.734827:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762348.734836:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762348/real 1789762348] req@ffff9018c66a6300 x1876699504186496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762348.734847:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504186496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762348.734860:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762348.734863:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762348.734866:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504186496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762348.734871:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504186624:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762348.734892:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762348.734907:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762348/real 1789762348] req@ffff9018c66a4700 x1876699504186624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762348.734924:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504186624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762348.734936:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762348.734960:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762348.734966:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504186624:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762348.734974:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504186752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762348.734998:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762348.735012:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762348/real 1789762348] req@ffff9018c66a6680 x1876699504186752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762348.735030:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504186752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762348.735098:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762348.735104:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762348.735111:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504186752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762348.735120:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504186880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762348.735150:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762348.735165:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762348/real 1789762348] req@ffff9018c66a5180 x1876699504186880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762348.735180:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504186880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762348.735190:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762348.735193:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762348.735197:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504186880:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762349.374238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762350.398239:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762351.422254:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762352.446166:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762353.470276:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762353.854211:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762353.854222:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762353.854228:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762353.854243:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2280 00000100:00080000:2.0:1789762353.854256:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762353.854263:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762353.854302:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504187008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762353.854327:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762353.854330:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762353.854332:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762353.854337:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2280 00000100:00080000:2.0:1789762353.854346:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762353.854350:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762353.854360:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762353.854369:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504187008:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762353.854375:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504187136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762353.854386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762353.854390:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762353.854391:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762353.854394:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2280 00000100:00080000:2.0:1789762353.854397:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762353.854400:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762353.854406:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762353.854421:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504187264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762353.854433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762353.854438:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762353.854440:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762353.854446:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2280 00000100:00080000:2.0:1789762353.854452:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762353.854458:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762353.854465:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789762353.854468:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762353.854484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504187392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762353.854504:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762353.854509:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789762353.854511:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762353/real 1789762353] req@ffff9018d0f24380 x1876699504187008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762353.854512:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762353.854522:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2280 00000100:00080000:2.0:1789762353.854528:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762353.854534:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762353.854535:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504187008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789762353.854543:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762353.854548:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762353.854552:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762353.854557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504187008:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762353.854562:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504187520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762353.854574:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762353.854578:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762353.854581:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762353.854585:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504187136:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789762353.854587:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2280 00000100:00080000:2.0:1789762353.854596:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762353.854602:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789762353.854608:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789762353.854609:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762353.854620:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762353/real 1789762353] req@ffff9018d0f27800 x1876699504187136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762353.854627:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504187648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762353.854634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504187136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762353.854654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762353.854657:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762353.854661:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504187136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762353.854667:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504187264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762353.854693:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762353.854719:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762353/real 1789762353] req@ffff9018d0f25180 x1876699504187264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762353.854745:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504187264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762353.854759:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762353.854764:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762353.854768:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504187264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762353.854777:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504187392:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762353.854812:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762353.854831:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762353/real 1789762353] req@ffff9018d0f24a80 x1876699504187392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762353.854840:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504187392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762353.854846:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762353.854848:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762353.854850:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504187392:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762353.854854:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504187520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762353.854865:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762353.854872:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762353/real 1789762353] req@ffff9018d0f25500 x1876699504187520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762353.854879:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504187520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762353.854884:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762353.854886:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762353.854887:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504187520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762353.854900:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504187648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762353.854931:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762353.854949:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762353/real 1789762353] req@ffff9018d0f27480 x1876699504187648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762353.854957:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504187648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762353.854962:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762353.854964:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762353.854966:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504187648:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762354.494193:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762355.518301:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762356.542178:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762357.566211:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762358.590197:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762358.974125:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762358.974159:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762358.974167:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762358.974183:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2285 00000100:00080000:2.0:1789762358.974200:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762358.974221:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762358.974292:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504187776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762358.974324:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762358.974328:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762358.974329:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762358.974333:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2285 00000100:00080000:2.0:1789762358.974336:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762358.974339:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762358.974350:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762358.974389:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504187904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762358.974401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762358.974405:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762358.974408:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762358.974413:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2285 00000100:00080000:2.0:1789762358.974417:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762358.974420:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762358.974426:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762358.974452:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504188032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762358.974468:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762358.974476:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762358.974489:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762358.974493:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504187776:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762358.974500:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2285 00000100:00080000:2.0:1789762358.974519:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762358.974526:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762358.974535:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762358.974567:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504188160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762358.974577:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762358.974580:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762358.974582:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762358.974594:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2285 00000100:00080000:2.0:1789762358.974600:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762358.974604:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762358.974610:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762358.974622:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504188288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762358.974634:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762358.974637:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762358.974638:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762358.974641:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2285 00000100:00080000:2.0:1789762358.974646:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762358.974651:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762358.974657:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762358.974668:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504188416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762358.974684:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504187904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762358.974736:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504188032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762358.974761:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504188160:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762358.974788:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504188288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762358.974812:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504188416:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762359.614279:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762360.638160:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762361.662227:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762362.046341:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:3.0:1789762362.046393:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762362.046421:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762362.046474:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762362.046492:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762362.046507:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762362.046522:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789762362.046523:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762358/real 1789762362] req@ffff9018d0f24700 x1876699504188032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1789762362.046533:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789762362.046565:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504188032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762362.046580:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762362.046587:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762362.046592:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504188032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762362.046603:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762358/real 1789762362] req@ffff9018d0f26680 x1876699504188160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762362.046617:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504188160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762362.046627:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762362.046631:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762362.046640:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504188160:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762362.046649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762358/real 1789762362] req@ffff9018d0f27480 x1876699504188288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762362.046662:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504188288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762362.046676:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762362.046679:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762362.046683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504188288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762362.046690:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762358/real 1789762362] req@ffff9018d0f25500 x1876699504188416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762362.046701:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504188416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762362.046708:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762362.046711:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762362.046714:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504188416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762362.046721:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762358/real 1789762362] req@ffff9018d0f24e00 x1876699504187776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762362.046734:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504187776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762362.046741:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762362.046744:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762362.046747:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504187776:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762362.046753:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762358/real 1789762362] req@ffff9018d0f24000 x1876699504187904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762362.046765:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504187904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762362.046774:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762362.046778:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762362.046781:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504187904:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762362.686184:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762363.710210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762364.094133:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762364.094144:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762364.094149:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762364.094161:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2290 00000100:00080000:2.0:1789762364.094172:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762364.094180:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762364.094221:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504188544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762364.094252:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762364.094256:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762364.094259:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762364.094263:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2290 00000100:00080000:2.0:1789762364.094267:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762364.094271:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762364.094283:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762364.094299:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504188672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762364.094309:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504188544:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762364.094316:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762364.094321:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762364.094323:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762364.094328:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2290 00000100:00080000:2.0:1789762364.094334:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762364.094339:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762364.094345:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762364.094360:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504188800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762364.094372:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762364.094376:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762364.094378:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762364.094383:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2290 00000100:00080000:2.0:1789762364.094387:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762364.094392:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789762364.094398:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789762364.094399:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762364.094416:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504188928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762364.094428:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762364.094432:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762364.094434:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762364.094438:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2290 00000100:00080000:2.0:1789762364.094442:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762364.094447:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762364.094453:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762364.094454:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762364/real 1789762364] req@ffff9018d0f24a80 x1876699504188544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762364.094472:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504189056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762364.094482:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504188544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762364.094517:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762364.094518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762364.094525:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762364.094526:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789762364.094529:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762364.094535:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504188544:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762364.094539:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2290 00000100:00080000:2.0:1789762364.094547:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762364.094552:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762364.094557:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762364.094570:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504189184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762364.094578:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504188672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762364.094611:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762364.094627:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762364/real 1789762364] req@ffff9018d0f25180 x1876699504188672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762364.094640:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504188672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762364.094648:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762364.094651:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762364.094655:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504188672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762364.094661:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504188800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762364.094682:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762364.094701:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762364/real 1789762364] req@ffff9018d0f27800 x1876699504188800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762364.094712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504188800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762364.094720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762364.094723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762364.094725:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504188800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762364.094731:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504188928:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762364.094750:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762364.094760:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762364/real 1789762364] req@ffff9018d0f24380 x1876699504188928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762364.094770:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504188928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762364.094777:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762364.094781:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762364.094784:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504188928:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762364.094789:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504189056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762364.094809:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762364.094820:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762364/real 1789762364] req@ffff9018d0f26a00 x1876699504189056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762364.094850:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504189056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762364.094859:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762364.094862:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762364.094866:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504189056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762364.094884:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504189184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762364.094907:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762364.094919:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762364/real 1789762364] req@ffff9018d0f25c00 x1876699504189184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762364.094933:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504189184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762364.094941:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762364.094944:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762364.094947:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504189184:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762364.734238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762365.758225:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762366.782239:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762367.806237:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762368.830176:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762369.214221:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762369.214238:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762369.214244:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762369.214259:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2295 00000100:00080000:3.0:1789762369.214276:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762369.214287:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762369.214371:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504189312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762369.214438:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762369.214443:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762369.214445:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762369.214449:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2295 00000100:00080000:3.0:1789762369.214454:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762369.214459:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762369.214478:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762369.214515:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504189440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762369.214531:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762369.214536:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762369.214540:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762369.214553:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2295 00000100:00080000:3.0:1789762369.214559:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762369.214564:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762369.214572:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762369.214604:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504189568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762369.214609:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504189312:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762369.214617:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762369.214621:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762369.214622:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762369.214626:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2295 00000100:00080000:3.0:1789762369.214629:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762369.214632:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762369.214637:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762369.214668:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504189696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762369.214681:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762369.214685:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762369.214688:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762369.214694:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2295 00000100:00080000:3.0:1789762369.214700:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762369.214705:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762369.214712:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762369.214738:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762369.214744:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504189824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762369.214760:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762369.214766:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762369.214769:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762369.214775:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2295 00000100:00080000:3.0:1789762369.214779:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762369.214784:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762369.214813:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762369.214858:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504189952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762369.214876:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762369/real 1789762369] req@ffff9018d070ce00 x1876699504189312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762369.214936:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504189312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762369.214974:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762369.214980:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762369.214998:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504189312:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762369.215009:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504189440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762369.215046:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762369.215111:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762369/real 1789762369] req@ffff9018d070ca80 x1876699504189440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762369.215141:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504189440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762369.215149:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762369.215152:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762369.215155:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504189440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762369.215205:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504189568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762369.215234:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762369.215256:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762369/real 1789762369] req@ffff9018d070fb80 x1876699504189568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762369.215270:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504189568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762369.215281:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762369.215285:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762369.215288:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504189568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762369.215296:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504189696:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762369.215316:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762369.215328:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762369/real 1789762369] req@ffff9018d070f480 x1876699504189696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762369.215341:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504189696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762369.215362:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762369.215366:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762369.215369:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504189696:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762369.215375:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504189824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762369.215394:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762369.215422:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762369/real 1789762369] req@ffff9018d070ed80 x1876699504189824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762369.215431:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504189824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762369.215438:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762369.215441:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762369.215444:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504189824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762369.215449:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504189952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762369.215474:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762369.215501:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762369/real 1789762369] req@ffff9018d070df80 x1876699504189952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762369.215515:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504189952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762369.215525:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762369.215528:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762369.215532:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504189952:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762369.854226:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762370.878246:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762371.902229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762372.926320:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762373.950193:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762374.334103:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762374.334112:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762374.334142:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762374.334152:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2300 00000100:00080000:3.0:1789762374.334163:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762374.334171:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762374.334211:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504190080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762374.334234:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762374.334238:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762374.334239:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762374.334242:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2300 00000100:00080000:3.0:1789762374.334246:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762374.334249:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762374.334258:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762374.334272:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504190208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762374.334285:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762374.334287:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504190080:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762374.334290:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762374.334293:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762374.334300:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2300 00000100:00080000:3.0:1789762374.334303:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762374.334306:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762374.334310:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762374.334321:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504190336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762374.334328:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762374.334331:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762374.334332:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762374.334335:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2300 00000100:00080000:3.0:1789762374.334338:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762374.334340:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762374.334344:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762374.334352:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504190464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762374.334359:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762374.334361:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762374.334362:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762374.334367:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2300 00000800:00000100:1.0:1789762374.334370:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762374.334371:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762374.334376:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762374.334379:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762374.334387:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504190592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762374.334395:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762374.334399:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762374.334401:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762374.334405:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2300 00000100:00080000:3.0:1789762374.334425:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762374.334432:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762374/real 1789762374] req@ffff9018d070d880 x1876699504190080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762374.334433:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762374.334444:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762374.334465:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504190720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762374.334467:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504190080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762374.334493:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762374.334500:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762374.334506:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504190080:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762374.334549:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504190208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762374.334585:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762374.334597:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762374/real 1789762374] req@ffff9018d070d180 x1876699504190208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762374.334611:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504190208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762374.334628:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762374.334632:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762374.334636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504190208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762374.334643:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504190336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762374.334673:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762374.334683:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762374/real 1789762374] req@ffff9018d070e680 x1876699504190336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762374.334697:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504190336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762374.334706:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762374.334722:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762374.334725:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504190336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762374.334732:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504190464:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762374.334767:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762374.334788:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762374/real 1789762374] req@ffff9018d070df80 x1876699504190464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762374.334802:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504190464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762374.334812:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762374.334816:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762374.334820:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504190464:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762374.334828:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504190592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762374.334849:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762374.334861:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762374/real 1789762374] req@ffff9018d070ed80 x1876699504190592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762374.334875:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504190592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762374.334884:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762374.334887:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762374.334891:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504190592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762374.334898:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504190720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762374.334916:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762374.334928:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762374/real 1789762374] req@ffff9018d070f480 x1876699504190720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762374.334941:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504190720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762374.334950:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762374.334954:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762374.334958:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504190720:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762374.974198:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762375.998171:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762377.022343:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762378.046157:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762379.070315:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762379.454154:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762379.454165:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762379.454169:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762379.454181:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2306 00000100:00080000:3.0:1789762379.454194:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762379.454202:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762379.454248:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504190848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762379.454276:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762379.454279:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762379.454280:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762379.454284:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2306 00000100:00080000:3.0:1789762379.454286:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762379.454289:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762379.454300:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762379.454311:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504190976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762379.454319:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762379.454322:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762379.454323:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762379.454326:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2306 00000100:00080000:3.0:1789762379.454328:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762379.454330:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762379.454336:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762379.454346:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504191104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762379.454383:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762379.454391:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762379.454396:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504190848:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762379.454396:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762379.454410:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2306 00000100:00080000:3.0:1789762379.454445:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762379.454451:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762379.454460:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762379.454491:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504191232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762379.454520:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762379.454524:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762379.454526:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762379.454530:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2306 00000100:00080000:3.0:1789762379.454544:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762379.454549:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762379.454554:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762379.454558:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762379.454575:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504191360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762379.454591:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762379.454597:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762379.454601:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762379.454609:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2306 00000100:00080000:3.0:1789762379.454618:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762379.454620:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762379/real 1789762379] req@ffff9018d070fb80 x1876699504190848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762379.454628:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762379.454645:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762379.454661:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504190848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762379.454666:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504191488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762379.454703:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762379.454732:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762379.454740:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504190848:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762379.454755:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504190976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762379.454791:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762379.454805:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762379/real 1789762379] req@ffff9018d070ca80 x1876699504190976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762379.454817:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504190976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762379.454825:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762379.454828:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762379.454831:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504190976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762379.454855:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504191104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762379.454872:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762379.454880:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762379/real 1789762379] req@ffff9018d070ce00 x1876699504191104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762379.454890:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504191104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762379.454913:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762379.454916:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762379.454919:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504191104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762379.454924:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504191232:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762379.454937:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762379.454945:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762379/real 1789762379] req@ffff9018d070e300 x1876699504191232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762379.454955:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504191232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762379.454962:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762379.454991:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762379.454993:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504191232:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762379.454998:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504191360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762379.455011:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762379.455018:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762379/real 1789762379] req@ffff9018d070c000 x1876699504191360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762379.455026:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504191360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762379.455032:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762379.455034:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762379.455037:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504191360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762379.455042:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504191488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762379.455093:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762379.455102:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762379/real 1789762379] req@ffff9018d070dc00 x1876699504191488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762379.455111:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504191488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762379.455118:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762379.455120:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762379.455122:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504191488:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762380.094305:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762381.118208:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762382.142234:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762383.166190:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762384.190180:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762384.574174:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762384.574183:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762384.574186:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762384.574195:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2311 00000100:00080000:0.0:1789762384.574205:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762384.574211:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762384.574246:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504191616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762384.574276:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762384.574281:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762384.574283:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762384.574288:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2311 00000100:00080000:0.0:1789762384.574292:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762384.574297:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762384.574309:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762384.574325:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504191744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762384.574333:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504191616:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762384.574348:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762384.574352:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762384.574354:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762384.574358:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2311 00000100:00080000:0.0:1789762384.574362:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762384.574365:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762384.574370:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762384.574381:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504191872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762384.574389:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762384.574392:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762384.574396:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789762384.574396:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762384.574404:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2311 00000100:00080000:0.0:1789762384.574406:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762384.574409:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762384.574413:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762384.574423:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504192000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762384.574441:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762384.574445:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762384.574450:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762384.574450:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762384/real 1789762384] req@ffff9018c8b98700 x1876699504191616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762384.574476:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2311 00000100:00080000:0.0:1789762384.574484:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762384.574492:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762384.574499:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504191616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789762384.574500:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762384.574516:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762384.574522:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762384.574523:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504192128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762384.574531:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504191616:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762384.574545:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762384.574548:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762384.574549:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762384.574553:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2311 00000100:00080000:0.0:1789762384.574557:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762384.574562:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504191744:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789762384.574563:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762384.574573:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762384.574585:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504192256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762384.574596:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762384.574635:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762384/real 1789762384] req@ffff9018c8b9bb80 x1876699504191744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762384.574647:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504191744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762384.574655:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762384.574658:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762384.574660:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504191744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762384.574665:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504191872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762384.574697:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762384.574707:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762384/real 1789762384] req@ffff9018c8b99880 x1876699504191872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762384.574716:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504191872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762384.574729:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762384.574731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762384.574734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504191872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762384.574739:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504192000:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762384.574768:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762384.574780:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762384/real 1789762384] req@ffff9018c8b98a80 x1876699504192000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762384.574795:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504192000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762384.574806:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762384.574810:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762384.574814:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504192000:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762384.574822:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504192128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762384.574844:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762384.574854:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762384/real 1789762384] req@ffff9018c8b99f80 x1876699504192128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762384.574863:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504192128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762384.574870:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762384.574872:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762384.574875:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504192128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762384.574890:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504192256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762384.574904:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762384.574911:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762384/real 1789762384] req@ffff9018c8b98700 x1876699504192256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762384.574936:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504192256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762384.574946:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762384.574949:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762384.574954:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504192256:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762385.214145:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762386.238222:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762387.262229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762388.286141:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762389.310203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762389.694101:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762389.694111:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762389.694115:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762389.694126:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2316 00000100:00080000:3.0:1789762389.694139:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762389.694152:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762389.694193:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504192384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762389.694217:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762389.694220:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762389.694221:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762389.694224:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2316 00000100:00080000:3.0:1789762389.694228:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762389.694231:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762389.694241:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762389.694252:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504192512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762389.694261:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762389.694264:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504192384:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762389.694265:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762389.694267:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762389.694275:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2316 00000100:00080000:3.0:1789762389.694278:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762389.694280:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762389.694284:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762389.694293:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504192640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762389.694299:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762389.694302:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762389.694304:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762389.694307:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2316 00000100:00080000:3.0:1789762389.694309:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762389.694311:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762389.694315:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762389.694324:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504192768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762389.694342:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762389.694343:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762389.694349:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762389.694352:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762389.694358:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2316 00000100:00080000:3.0:1789762389.694365:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762389.694370:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762389.694376:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762389.694389:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504192896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762389.694393:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762389/real 1789762389] req@ffff9018d070d500 x1876699504192384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762389.694409:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762389.694413:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762389.694415:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762389.694418:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2316 00000100:00080000:3.0:1789762389.694422:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762389.694423:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504192384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762389.694432:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789762389.694438:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789762389.694440:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762389.694444:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762389.694450:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504192384:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762389.694459:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504193024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762389.694474:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504192512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762389.694493:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762389.694504:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762389/real 1789762389] req@ffff9018d070f800 x1876699504192512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762389.694513:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504192512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762389.694520:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762389.694522:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762389.694525:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504192512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762389.694530:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504192640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762389.694546:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762389.694553:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762389/real 1789762389] req@ffff9018d070f100 x1876699504192640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762389.694562:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504192640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762389.694568:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762389.694570:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762389.694572:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504192640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762389.694576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504192768:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762389.694589:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762389.694595:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762389/real 1789762389] req@ffff9018d070dc00 x1876699504192768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762389.694602:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504192768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762389.694608:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762389.694610:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762389.694612:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504192768:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762389.694616:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504192896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762389.694630:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762389.694655:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762389/real 1789762389] req@ffff9018d070c000 x1876699504192896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762389.694667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504192896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762389.694675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762389.694678:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762389.694680:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504192896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762389.694700:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504193024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762389.694715:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762389.694723:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762389/real 1789762389] req@ffff9018d070e300 x1876699504193024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762389.694732:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504193024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762389.694738:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762389.694740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762389.694743:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504193024:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762390.334156:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762391.358249:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762392.382176:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762393.406194:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762394.430179:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762394.814152:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762394.814163:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762394.814181:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762394.814193:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2321 00000100:00080000:3.0:1789762394.814205:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762394.814212:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762394.814256:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504193152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762394.814285:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762394.814289:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762394.814292:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762394.814297:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2321 00000100:00080000:3.0:1789762394.814300:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762394.814305:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762394.814318:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762394.814334:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504193280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762394.814361:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762394.814365:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762394.814368:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762394.814372:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2321 00000100:00080000:3.0:1789762394.814375:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762394.814380:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762394.814383:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504193152:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789762394.814385:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762394.814397:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504193408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762394.814426:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762394.814429:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762394.814431:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762394.814434:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2321 00000100:00080000:3.0:1789762394.814437:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762394.814439:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762394.814444:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762394.814453:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504193536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762394.814460:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762394.814462:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762394.814464:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762394.814466:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2321 00000100:00080000:3.0:1789762394.814471:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762394.814473:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762394.814476:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762394.814484:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762394.814486:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504193664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762394.814494:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762394.814497:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762394.814498:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762394.814501:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2321 00000100:00080000:3.0:1789762394.814514:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762394.814517:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762394.814522:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762394.814530:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762394/real 1789762394] req@ffff9018d070ce00 x1876699504193152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762394.814533:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504193792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762394.814571:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504193152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762394.814582:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762394.814586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762394.814590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504193152:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762394.814616:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504193280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762394.814651:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762394.814661:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762394/real 1789762394] req@ffff9018d070ca80 x1876699504193280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762394.814669:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504193280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762394.814675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762394.814677:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762394.814679:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504193280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762394.814701:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504193408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762394.814711:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762394.814717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762394/real 1789762394] req@ffff9018d070fb80 x1876699504193408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762394.814724:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504193408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762394.814729:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762394.814731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762394.814733:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504193408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762394.814736:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504193536:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762394.814746:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762394.814752:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762394/real 1789762394] req@ffff9018d070f480 x1876699504193536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762394.814760:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504193536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762394.814765:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762394.814767:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762394.814768:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504193536:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762394.814771:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504193664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762394.814781:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762394.814787:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762394/real 1789762394] req@ffff9018d070ed80 x1876699504193664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762394.814794:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504193664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762394.814799:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762394.814800:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762394.814802:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504193664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762394.814805:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504193792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762394.814815:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762394.814820:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762394/real 1789762394] req@ffff9018d070df80 x1876699504193792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762394.814826:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504193792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762394.814831:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762394.814834:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762394.814836:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504193792:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762395.454167:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762396.478176:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762397.502205:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762398.526185:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762399.550231:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762399.934143:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762399.934151:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762399.934155:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762399.934164:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2326 00000100:00080000:3.0:1789762399.934174:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762399.934196:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762399.934230:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504193920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762399.934269:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762399.934272:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762399.934274:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762399.934278:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2326 00000100:00080000:3.0:1789762399.934280:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762399.934285:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762399.934293:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762399.934306:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504194048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762399.934316:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762399.934318:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762399.934320:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762399.934326:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2326 00000100:00100000:1.0:1789762399.934330:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504193920:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762399.934331:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762399.934343:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762399.934347:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762399.934359:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504194176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762399.934366:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762399.934370:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762399.934371:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762399.934373:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2326 00000100:00080000:3.0:1789762399.934375:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762399.934379:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762399.934382:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762399.934391:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504194304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762399.934397:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762399.934399:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762399.934400:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762399.934405:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2326 00000100:00080000:3.0:1789762399.934410:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762399.934413:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762399.934417:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762399.934423:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762399.934428:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504194432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762399.934438:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762399.934440:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762399.934442:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762399.934445:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2326 00000100:00080000:3.0:1789762399.934449:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762399.934452:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762399.934456:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762399.934465:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504194560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762399.934478:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762399/real 1789762399] req@ffff9018d070e680 x1876699504193920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762399.934495:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504193920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762399.934505:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762399.934508:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762399.934512:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504193920:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762399.934535:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504194048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762399.934552:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762399.934576:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762399/real 1789762399] req@ffff9018d070d180 x1876699504194048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762399.934585:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504194048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762399.934591:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762399.934593:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762399.934595:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504194048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762399.934600:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504194176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762399.934615:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762399.934622:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762399/real 1789762399] req@ffff9018d070d880 x1876699504194176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762399.934629:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504194176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762399.934634:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762399.934636:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762399.934638:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504194176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762399.934642:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504194304:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762399.934654:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762399.934660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762399/real 1789762399] req@ffff9018d070df80 x1876699504194304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762399.934667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504194304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762399.934672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762399.934674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762399.934676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504194304:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762399.934679:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504194432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762399.934689:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762399.934695:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762399/real 1789762399] req@ffff9018d070ed80 x1876699504194432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762399.934703:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504194432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762399.934708:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762399.934709:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762399.934711:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504194432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762399.934715:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504194560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762399.934725:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762399.934730:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762399/real 1789762399] req@ffff9018d070f480 x1876699504194560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762399.934737:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504194560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762399.934743:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762399.934746:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762399.934750:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504194560:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762400.574179:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762401.598169:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762402.622172:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762403.646179:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762404.670208:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762405.054085:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762405.054099:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762405.054104:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762405.054118:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2331 00000100:00080000:3.0:1789762405.054131:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762405.054137:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762405.054179:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504194688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762405.054204:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762405.054208:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762405.054209:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762405.054213:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2331 00000100:00080000:3.0:1789762405.054215:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762405.054218:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762405.054228:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762405.054238:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504194816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762405.054243:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504194688:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762405.054252:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762405.054256:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762405.054258:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762405.054261:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2331 00000100:00080000:3.0:1789762405.054264:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762405.054267:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762405.054270:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762405.054278:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504194944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762405.054285:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762405.054288:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762405.054289:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762405.054292:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2331 00000100:00080000:3.0:1789762405.054295:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762405.054299:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762405.054303:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762405.054318:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504195072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762405.054326:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762405.054331:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762405.054332:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762405.054335:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762405.054340:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2331 00000100:00080000:3.0:1789762405.054344:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762405.054347:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762405.054351:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762405.054360:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504195200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762405.054367:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762405.054370:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762405.054371:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762405.054376:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2331 00000100:00100000:1.0:1789762405.054377:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762405/real 1789762405] req@ffff9018d070fb80 x1876699504194688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762405.054379:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762405.054385:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762405.054392:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762405.054403:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504195328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762405.054406:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504194688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762405.054456:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762405.054460:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762405.054463:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504194688:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762405.054489:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504194816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762405.054506:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762405.054515:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762405/real 1789762405] req@ffff9018d070ca80 x1876699504194816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762405.054540:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504194816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762405.054546:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762405.054548:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762405.054550:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504194816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762405.054555:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504194944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762405.054574:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762405.054585:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762405/real 1789762405] req@ffff9018d070ce00 x1876699504194944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762405.054598:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504194944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762405.054606:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762405.054610:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762405.054613:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504194944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762405.054620:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504195072:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762405.054640:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762405.054651:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762405/real 1789762405] req@ffff9018d070e300 x1876699504195072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762405.054662:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504195072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762405.054669:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762405.054673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762405.054677:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504195072:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762405.054683:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504195200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762405.054705:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762405.054717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762405/real 1789762405] req@ffff9018d070c000 x1876699504195200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762405.054729:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504195200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762405.054735:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762405.054737:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762405.054740:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504195200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762405.054744:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504195328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762405.054758:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762405.054766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762405/real 1789762405] req@ffff9018d070dc00 x1876699504195328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762405.054773:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504195328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762405.054778:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762405.054780:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762405.054781:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504195328:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762405.694141:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762406.718171:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762407.742175:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762408.766227:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762409.790188:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762410.174113:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762410.174124:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762410.174128:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762410.174139:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2336 00000100:00080000:3.0:1789762410.174151:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762410.174158:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762410.174202:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504195456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762410.174226:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762410.174230:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762410.174231:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762410.174234:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2336 00000100:00080000:3.0:1789762410.174237:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762410.174239:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762410.174249:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762410.174259:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504195584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762410.174267:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762410.174270:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762410.174271:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762410.174278:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2336 00000100:00080000:3.0:1789762410.174281:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762410.174282:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504195456:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762410.174285:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762410.174291:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762410.174299:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504195712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762410.174305:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762410.174307:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762410.174309:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762410.174312:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2336 00000100:00080000:3.0:1789762410.174314:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762410.174316:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762410.174319:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762410.174328:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504195840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762410.174334:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762410.174337:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762410.174338:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762410.174341:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2336 00000100:00080000:3.0:1789762410.174344:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762410.174347:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762410.174349:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762410.174356:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504195968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762410.174363:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762410.174365:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762410.174367:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762410.174369:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2336 00000100:00080000:3.0:1789762410.174372:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762410.174375:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762410.174376:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762410.174381:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762410.174389:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504196096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762410.174433:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762410/real 1789762410] req@ffff9018d070f100 x1876699504195456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762410.174457:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504195456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762410.174469:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762410.174473:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762410.174478:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504195456:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762410.174522:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504195584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762410.174549:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762410.174564:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762410/real 1789762410] req@ffff9018d070f800 x1876699504195584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762410.174577:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504195584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762410.174604:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762410.174607:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762410.174612:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504195584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762410.174619:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504195712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762410.174643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762410.174655:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762410/real 1789762410] req@ffff9018d070d500 x1876699504195712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762410.174667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504195712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762410.174676:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762410.174680:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762410.174683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504195712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762410.174690:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504195840:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762410.174713:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762410.174725:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762410/real 1789762410] req@ffff9018d070dc00 x1876699504195840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762410.174737:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504195840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762410.174747:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762410.174750:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762410.174754:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504195840:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762410.174761:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504195968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762410.174779:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762410.174789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762410/real 1789762410] req@ffff9018d070c000 x1876699504195968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762410.174800:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504195968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762410.174809:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762410.174812:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762410.174815:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504195968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762410.174821:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504196096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762410.174836:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762410.174846:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762410/real 1789762410] req@ffff9018d070e300 x1876699504196096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762410.174858:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504196096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762410.174867:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762410.174870:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762410.174873:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504196096:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762410.814252:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762411.838170:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762412.862137:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762413.886289:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762414.910169:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762415.294132:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762415.294142:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762415.294147:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762415.294158:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2341 00000100:00080000:0.0:1789762415.294170:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762415.294177:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762415.294215:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504196224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762415.294238:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762415.294241:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762415.294243:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762415.294246:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2341 00000100:00080000:0.0:1789762415.294250:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762415.294253:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762415.294260:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762415.294272:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504196352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762415.294283:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762415.294287:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789762415.294289:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504196224:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762415.294290:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762415.294300:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2341 00000100:00080000:0.0:1789762415.294305:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762415.294309:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762415.294312:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762415.294322:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504196480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762415.294331:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762415.294335:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762415.294337:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762415.294341:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2341 00000100:00080000:0.0:1789762415.294346:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762415.294351:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762415.294358:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762415.294372:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504196608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762415.294383:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762415.294386:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762415.294388:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762415.294392:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2341 00000800:00000100:1.0:1789762415.294396:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762415.294398:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762415.294405:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762415.294411:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762415.294421:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504196736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762415.294429:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762415.294431:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762415.294433:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762415.294435:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2341 00000100:00080000:0.0:1789762415.294437:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762415.294440:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762415.294457:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762415.294467:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504196864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762415.294473:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762415/real 1789762415] req@ffff9018c8b9b800 x1876699504196224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762415.294507:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504196224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762415.294519:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762415.294525:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762415.294529:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504196224:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762415.294561:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504196352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762415.294592:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762415.294618:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762415/real 1789762415] req@ffff9018c8b9b100 x1876699504196352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762415.294628:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504196352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762415.294634:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762415.294637:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762415.294639:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504196352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762415.294644:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504196480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762415.294660:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762415.294667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762415/real 1789762415] req@ffff9018c8b9b480 x1876699504196480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762415.294675:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504196480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762415.294684:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762415.294687:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762415.294689:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504196480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762415.294693:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504196608:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762415.294704:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762415.294710:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762415/real 1789762415] req@ffff9018c8b98000 x1876699504196608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762415.294717:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504196608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762415.294723:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762415.294726:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762415.294730:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504196608:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762415.294737:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504196736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762415.294757:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762415.294769:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762415/real 1789762415] req@ffff9018c8b9aa00 x1876699504196736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762415.294782:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504196736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762415.294801:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762415.294805:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762415.294809:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504196736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762415.294815:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504196864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762415.294837:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762415.294849:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762415/real 1789762415] req@ffff9018c8b98e00 x1876699504196864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762415.294872:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504196864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762415.294881:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762415.294885:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762415.294888:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504196864:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762415.934214:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762416.958172:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762417.982206:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762419.006197:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762420.030248:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762420.414150:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762420.414160:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762420.414164:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762420.414179:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2346 00000100:00080000:0.0:1789762420.414193:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762420.414202:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762420.414251:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504196992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762420.414277:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762420.414280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762420.414281:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762420.414288:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2346 00000100:00080000:0.0:1789762420.414291:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762420.414295:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762420.414304:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762420.414313:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504197120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762420.414320:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762420.414323:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762420.414324:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762420.414327:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2346 00000100:00080000:0.0:1789762420.414329:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762420.414331:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762420.414334:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762420.414343:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504197248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762420.414344:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504196992:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762420.414359:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762420.414361:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762420.414362:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762420.414365:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2346 00000100:00080000:0.0:1789762420.414367:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762420.414369:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762420.414373:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762420.414386:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504197376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762420.414395:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762420.414398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762420.414399:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762420.414402:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2346 00000100:00080000:0.0:1789762420.414417:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762420.414420:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762420.414424:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762420.414433:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504197504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762420.414440:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762420.414443:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762420.414444:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762420.414447:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2346 00000100:00080000:0.0:1789762420.414449:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762420.414451:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762420.414454:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762420.414460:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762420.414465:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504197632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762420.414516:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762420/real 1789762420] req@ffff9018c8b9a680 x1876699504196992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762420.414538:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504196992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762420.414552:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762420.414557:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762420.414561:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504196992:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762420.414569:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504197120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762420.414590:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762420.414600:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762420/real 1789762420] req@ffff9018c8b98380 x1876699504197120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762420.414608:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504197120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762420.414614:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762420.414616:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762420.414619:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504197120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762420.414641:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504197248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762420.414653:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762420.414659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762420/real 1789762420] req@ffff9018c8b99c00 x1876699504197248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762420.414667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504197248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762420.414672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762420.414674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762420.414677:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504197248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762420.414683:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504197376:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762420.414706:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762420.414718:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762420/real 1789762420] req@ffff9018c8b99500 x1876699504197376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762420.414730:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504197376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762420.414739:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762420.414741:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762420.414743:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504197376:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762420.414747:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504197504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762420.414763:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762420.414771:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762420/real 1789762420] req@ffff9018c8b9ad80 x1876699504197504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762420.414778:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504197504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762420.414783:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762420.414785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762420.414787:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504197504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762420.414790:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504197632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762420.414800:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762420.414806:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762420/real 1789762420] req@ffff9018c8b9a300 x1876699504197632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762420.414812:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504197632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762420.414817:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762420.414819:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762420.414821:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504197632:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762421.054191:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762422.078260:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762423.102291:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762424.126215:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762425.150322:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762425.534263:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762425.534275:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762425.534281:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762425.534294:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2352 00000100:00080000:1.0:1789762425.534310:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762425.534321:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762425.534417:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504197760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762425.534457:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762425.534462:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762425.534466:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762425.534473:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2352 00000100:00080000:1.0:1789762425.534478:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762425.534485:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762425.534502:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762425.534528:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504197888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762425.534546:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504197760:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762425.534552:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762425.534561:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762425.534565:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762425.534575:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2352 00000100:00080000:1.0:1789762425.534582:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762425.534591:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762425.534602:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762425.534626:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504198016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762425.534647:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762425.534654:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762425.534658:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762425.534666:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2352 00000100:00080000:1.0:1789762425.534674:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762425.534684:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762425.534697:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762425.534725:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504197888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762425.534730:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504198144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762425.534760:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762425.534765:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762425.534768:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762425.534776:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2352 00000100:00080000:1.0:1789762425.534787:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762425.534798:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762425.534807:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762425.534808:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504198016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762425.534837:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504198272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762425.534852:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762425.534858:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762425.534860:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762425.534866:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2352 00000100:00080000:1.0:1789762425.534873:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762425.534879:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762425.534886:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504198144:192.168.203.158@tcp:8:kworker.0 00000008:00100000:1.0:1789762425.534890:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762425.534919:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504198400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762425.534945:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504198272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762425.534997:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504198400:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762426.174388:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762427.198214:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762428.222184:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762428.606421:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789762428.606480:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762428.606503:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762428.606548:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762428.606555:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762428.606563:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762428.606571:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762428.606578:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789762428.606602:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762425/real 1789762428] req@ffff9018c66a5880 x1876699504198400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762428.606642:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504198400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762428.606656:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762428.606663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762428.606668:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504198400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762428.606677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762425/real 1789762428] req@ffff9018c66a4380 x1876699504198144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762428.606690:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504198144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762428.606699:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762428.606703:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762428.606706:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504198144:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762428.606713:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762425/real 1789762428] req@ffff9018c66a6a00 x1876699504198272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762428.606732:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504198272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762428.606751:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762428.606754:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762428.606758:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504198272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762428.606764:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762425/real 1789762428] req@ffff9018c66a7480 x1876699504197888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762428.606777:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504197888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762428.606784:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762428.606788:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762428.606791:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504197888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762428.606797:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762425/real 1789762428] req@ffff9018c66a4a80 x1876699504198016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762428.606808:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504198016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762428.606816:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762428.606820:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762428.606823:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504198016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762428.606830:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762425/real 1789762428] req@ffff9018c66a7100 x1876699504197760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762428.606844:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504197760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762428.606851:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762428.606855:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762428.606858:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504197760:192.168.203.158@tcp:250:kworker.0 00010000:00010000:3.0:1789762429.246253:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762430.270211:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762430.654210:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762430.654225:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762430.654231:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762430.654247:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2357 00000100:00080000:0.0:1789762430.654264:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762430.654274:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762430.654361:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504198528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762430.654404:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762430.654412:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762430.654417:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762430.654433:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2357 00000100:00080000:0.0:1789762430.654442:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762430.654451:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762430.654469:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762430.654474:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504198528:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762430.654495:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504198656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762430.654525:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762430.654531:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762430.654534:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762430.654540:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2357 00000100:00080000:0.0:1789762430.654561:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762430.654568:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762430.654576:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762430.654587:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762430.654595:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504198784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762430.654611:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762430.654617:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762430.654620:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762430.654626:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2357 00000100:00080000:0.0:1789762430.654631:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762430.654638:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762430.654647:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762430.654666:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504198912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762430.654667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762430/real 1789762430] req@ffff9018c8b9a300 x1876699504198528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762430.654685:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762430.654693:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762430.654695:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762430.654701:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2357 00000100:00080000:0.0:1789762430.654709:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762430.654710:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504198528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762430.654716:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762430.654725:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762430.654732:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762430.654741:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762430.654742:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504199040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762430.654749:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504198528:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762430.654760:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762430.654766:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762430.654769:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762430.654775:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2357 00000100:00080000:0.0:1789762430.654781:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762430.654787:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762430.654794:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504198656:192.168.203.158@tcp:38:kworker.0 00000008:00100000:0.0:1789762430.654795:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762430.654811:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504199168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762430.654839:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762430.654865:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762430/real 1789762430] req@ffff9018c8b9ad80 x1876699504198656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762430.654891:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504198656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762430.654909:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762430.654916:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762430.654934:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504198656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762430.654948:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504198784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762430.654994:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762430.655018:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762430/real 1789762430] req@ffff9018c8b99500 x1876699504198784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762430.655090:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504198784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762430.655108:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762430.655115:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762430.655122:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504198784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762430.655135:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504198912:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762430.655186:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762430.655221:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762430/real 1789762430] req@ffff9018c8b99c00 x1876699504198912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762430.655249:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504198912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762430.655265:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762430.655272:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762430.655279:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504198912:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762430.655292:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504199040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762430.655380:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762430.655403:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762430/real 1789762430] req@ffff9018c8b98380 x1876699504199040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762430.655425:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504199040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762430.655438:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762430.655443:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762430.655448:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504199040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762430.655482:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504199168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762430.655532:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762430.655555:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762430/real 1789762430] req@ffff9018c8b9a680 x1876699504199168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762430.655577:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504199168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762430.655592:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762430.655598:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762430.655604:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504199168:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762431.294266:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762432.318297:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762433.342272:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762434.366282:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762435.390453:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762435.774205:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762435.774237:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762435.774255:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762435.774270:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2362 00000100:00080000:0.0:1789762435.774287:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762435.774356:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762435.774432:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504199296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762435.774478:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762435.774485:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762435.774489:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762435.774495:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2362 00000100:00080000:0.0:1789762435.774502:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762435.774511:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762435.774529:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762435.774559:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504199296:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762435.774568:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504199424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762435.774585:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762435.774589:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762435.774591:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762435.774596:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2362 00000100:00080000:0.0:1789762435.774601:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762435.774605:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762435.774611:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762435.774638:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504199552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762435.774648:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762435.774653:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762435.774655:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762435.774658:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2362 00000100:00080000:0.0:1789762435.774666:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762435.774670:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762435.774677:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762435.774687:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504199680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762435.774697:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762435.774701:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762435.774706:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789762435.774709:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762435.774714:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2362 00000100:00080000:0.0:1789762435.774720:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762435.774725:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762435.774730:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762435.774741:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504199808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762435.774750:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762435.774754:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762435.774756:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762435.774760:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2362 00000100:00080000:0.0:1789762435.774764:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762435.774767:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762435.774773:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762435.774792:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504199936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762435.774792:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762435/real 1789762435] req@ffff9018c8b98e00 x1876699504199296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762435.774850:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504199296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762435.774870:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762435.774878:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762435.774885:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504199296:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762435.774939:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504199424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762435.774993:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762435.775010:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762435/real 1789762435] req@ffff9018c8b9aa00 x1876699504199424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762435.775030:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504199424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762435.775062:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762435.775068:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762435.775075:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504199424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762435.775142:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504199552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762435.775175:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762435.775210:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762435/real 1789762435] req@ffff9018c8b98000 x1876699504199552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762435.775229:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504199552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762435.775257:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762435.775262:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762435.775268:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504199552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762435.775277:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504199680:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762435.775327:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762435.775369:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762435/real 1789762435] req@ffff9018c8b9b480 x1876699504199680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762435.775388:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504199680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762435.775412:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762435.775417:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762435.775423:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504199680:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762435.775435:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504199808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762435.775467:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762435.775493:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762435/real 1789762435] req@ffff9018c8b9b100 x1876699504199808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762435.775510:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504199808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762435.775522:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762435.775526:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762435.775532:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504199808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762435.775540:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504199936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762435.775567:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762435.775581:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762435/real 1789762435] req@ffff9018c8b9b800 x1876699504199936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762435.775597:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504199936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762435.775609:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762435.775614:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762435.775620:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504199936:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762436.414220:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762437.438417:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762438.462217:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762439.486341:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762440.510278:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762440.894200:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762440.894214:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762440.894220:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762440.894234:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2367 00000100:00080000:0.0:1789762440.894285:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762440.894307:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762440.894370:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504200064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762440.894399:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762440.894402:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762440.894403:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762440.894407:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2367 00000100:00080000:0.0:1789762440.894410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762440.894413:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762440.894424:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762440.894435:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504200192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762440.894443:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762440.894446:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762440.894448:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762440.894450:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2367 00000100:00080000:0.0:1789762440.894453:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762440.894456:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762440.894460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762440.894462:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504200064:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762440.894473:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504200320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762440.894482:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762440.894485:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762440.894486:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762440.894489:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2367 00000100:00080000:0.0:1789762440.894502:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762440.894505:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762440.894509:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762440.894518:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504200448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762440.894526:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762440.894529:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762440.894530:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762440.894532:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2367 00000100:00080000:0.0:1789762440.894547:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762440.894551:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762440.894554:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762440.894561:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762440.894567:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504200576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762440.894575:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762440.894578:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762440.894580:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762440.894583:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2367 00000100:00080000:0.0:1789762440.894598:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762440.894600:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762440.894604:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762440.894612:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504200704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762440.894629:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762440/real 1789762440] req@ffff9018c8b98700 x1876699504200064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762440.894650:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504200064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762440.894660:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762440.894664:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762440.894668:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504200064:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762440.894674:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504200192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762440.894691:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762440.894701:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762440/real 1789762440] req@ffff9018c8b99f80 x1876699504200192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762440.894729:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504200192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762440.894737:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762440.894740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762440.894743:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504200192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762440.894782:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504200320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762440.894798:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762440.894806:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762440/real 1789762440] req@ffff9018c8b98a80 x1876699504200320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762440.894816:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504200320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762440.894823:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762440.894825:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762440.894828:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504200320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762440.894833:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504200448:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762440.894847:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762440.894855:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762440/real 1789762440] req@ffff9018c8b99880 x1876699504200448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762440.894865:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504200448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762440.894873:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762440.894876:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762440.894878:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504200448:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762440.894883:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504200576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762440.894896:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762440.894904:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762440/real 1789762440] req@ffff9018c8b9bb80 x1876699504200576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762440.894914:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504200576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762440.894921:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762440.894924:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762440.894926:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504200576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762440.894931:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504200704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762440.894960:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762440.894966:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762440/real 1789762440] req@ffff9018c8b9b800 x1876699504200704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762440.894973:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504200704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762440.894979:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762440.894981:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762440.894983:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504200704:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762441.534439:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762442.558233:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762443.582255:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762444.606264:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762445.630302:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762446.014332:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762446.014349:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762446.014356:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762446.014367:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2372 00000100:00080000:1.0:1789762446.014382:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762446.014392:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762446.014441:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504200832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762446.014498:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762446.014503:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762446.014506:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762446.014511:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2372 00000100:00080000:1.0:1789762446.014515:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762446.014520:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762446.014531:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762446.014571:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504200960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762446.014584:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762446.014587:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762446.014589:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762446.014592:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2372 00000100:00080000:1.0:1789762446.014595:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762446.014604:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762446.014607:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504200832:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789762446.014627:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762446.014650:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504201088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762446.014663:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762446.014668:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762446.014671:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762446.014675:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2372 00000100:00080000:1.0:1789762446.014680:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762446.014685:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762446.014691:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762446.014704:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504201216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762446.014716:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762446.014720:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762446.014726:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762446.014729:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762446.014734:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2372 00000100:00080000:1.0:1789762446.014742:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762446.014747:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762446.014753:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762446.014775:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504201344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762446.014791:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762446.014799:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789762446.014801:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762446/real 1789762446] req@ffff9018c66a7100 x1876699504200832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762446.014814:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762446.014826:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2372 00000100:00080000:1.0:1789762446.014835:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762446.014844:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762446.014849:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762446.014869:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504201472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762446.014900:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504200832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762446.014920:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762446.014927:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762446.014940:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504200832:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762446.014951:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504200960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762446.015014:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762446.015036:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762446/real 1789762446] req@ffff9018c66a4a80 x1876699504200960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762446.015095:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504200960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762446.015133:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762446.015138:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762446.015144:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504200960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762446.015199:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504201088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762446.015276:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762446.015290:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762446/real 1789762446] req@ffff9018c66a7480 x1876699504201088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762446.015303:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504201088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762446.015333:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762446.015340:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762446.015347:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504201088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762446.015359:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504201216:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762446.015394:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762446.015414:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762446/real 1789762446] req@ffff9018c66a6a00 x1876699504201216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762446.015439:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504201216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762446.015456:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762446.015462:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762446.015486:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504201216:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762446.015497:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504201344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762446.015533:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762446.015567:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762446/real 1789762446] req@ffff9018c66a4380 x1876699504201344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762446.015583:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504201344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762446.015594:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762446.015598:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762446.015603:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504201344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762446.015611:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504201472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762446.015658:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762446.015680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762446/real 1789762446] req@ffff9018c66a5880 x1876699504201472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762446.015700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504201472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762446.015710:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762446.015714:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762446.015718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504201472:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762446.654240:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762447.678364:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762448.702336:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762449.726320:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762450.750368:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762451.134259:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762451.134275:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762451.134298:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762451.134314:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2377 00000100:00080000:1.0:1789762451.134331:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762451.134342:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762451.134411:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504201600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762451.134452:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762451.134458:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762451.134461:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762451.134466:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2377 00000100:00080000:1.0:1789762451.134470:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762451.134477:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762451.134492:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762451.134532:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504201728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762451.134557:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762451.134564:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762451.134567:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762451.134575:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2377 00000100:00080000:1.0:1789762451.134583:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762451.134592:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504201600:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762451.134596:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762451.134620:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762451.134652:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504201856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762451.134672:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762451.134679:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762451.134682:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762451.134689:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2377 00000100:00080000:1.0:1789762451.134699:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762451.134707:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789762451.134716:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:1.0:1789762451.134718:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762451.134766:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504201984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762451.134784:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762451.134790:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762451.134793:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762451.134800:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2377 00000100:00080000:1.0:1789762451.134807:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762451.134814:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762451.134824:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762451/real 1789762451] req@ffff9018c66a5180 x1876699504201600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762451.134827:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762451.134854:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504202112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762451.134902:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762451.134909:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789762451.134912:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504201600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762451.134914:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762451.134940:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2377 00000100:00080000:1.0:1789762451.134977:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762451.134983:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762451.134984:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789762451.135005:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:1.0:1789762451.135007:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762451.135046:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504201600:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762451.135057:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504202240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762451.135215:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504201728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762451.135252:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762451.135268:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762451/real 1789762451] req@ffff9018c66a6680 x1876699504201728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762451.135283:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504201728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762451.135292:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762451.135296:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762451.135299:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504201728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762451.135315:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504201856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762451.135333:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762451.135342:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762451/real 1789762451] req@ffff9018c66a4700 x1876699504201856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762451.135355:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504201856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762451.135363:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762451.135366:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762451.135369:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504201856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762451.135375:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504201984:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762451.135393:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762451.135403:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762451/real 1789762451] req@ffff9018c66a6300 x1876699504201984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762451.135414:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504201984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762451.135422:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762451.135425:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762451.135428:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504201984:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762451.135433:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504202112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762451.135450:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762451.135459:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762451/real 1789762451] req@ffff9018c66a4e00 x1876699504202112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762451.135470:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504202112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762451.135490:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762451.135495:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762451.135500:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504202112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762451.135538:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504202240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762451.135577:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762451.135601:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762451/real 1789762451] req@ffff9018c66a5500 x1876699504202240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762451.135620:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504202240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762451.135630:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762451.135634:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762451.135639:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504202240:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762451.774315:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762452.798381:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762453.822184:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762454.846327:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762455.870252:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762456.254112:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762456.254122:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762456.254126:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762456.254150:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2382 00000100:00080000:1.0:1789762456.254160:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762456.254168:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762456.254203:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504202368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762456.254230:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762456.254234:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762456.254236:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762456.254244:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2382 00000100:00080000:1.0:1789762456.254248:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762456.254252:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762456.254263:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762456.254276:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504202496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762456.254282:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504202368:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762456.254294:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762456.254299:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762456.254301:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762456.254306:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2382 00000100:00080000:1.0:1789762456.254310:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762456.254314:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762456.254320:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762456.254331:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504202624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762456.254341:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762456.254346:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762456.254350:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762456.254354:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762456.254359:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2382 00000100:00080000:1.0:1789762456.254364:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762456.254369:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762456.254376:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762456.254384:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762456/real 1789762456] req@ffff9018c66a7800 x1876699504202368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762456.254391:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504202752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762456.254403:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504202368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762456.254407:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762456.254418:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762456.254423:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762456.254428:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762456.254432:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2382 00000100:00080000:0.0:1789762456.254436:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789762456.254441:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762456.254443:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504202368:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762456.254450:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762456.254457:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762456.254470:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504202880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762456.254472:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504202496:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789762456.254489:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762456.254494:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762456.254496:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762456.254500:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2382 00000100:00080000:1.0:1789762456.254504:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762456.254508:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762456.254511:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762456.254517:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762456.254526:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762456/real 1789762456] req@ffff9018c66a7b80 x1876699504202496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762456.254530:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504203008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762456.254553:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504202496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762456.254560:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762456.254563:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762456.254567:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504202496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762456.254573:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504202624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762456.254589:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762456.254595:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762456/real 1789762456] req@ffff9018c66a4000 x1876699504202624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762456.254603:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504202624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762456.254608:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762456.254610:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762456.254612:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504202624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762456.254616:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504202752:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762456.254626:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762456.254632:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762456/real 1789762456] req@ffff9018c66a5f80 x1876699504202752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762456.254640:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504202752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762456.254647:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762456.254650:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762456.254653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504202752:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762456.254671:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504202880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762456.254692:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762456.254704:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762456/real 1789762456] req@ffff9018c66a6d80 x1876699504202880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762456.254713:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504202880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762456.254718:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762456.254721:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762456.254723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504202880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762456.254726:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504203008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762456.254739:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762456.254757:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762456/real 1789762456] req@ffff9018c66a5c00 x1876699504203008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762456.254765:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504203008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762456.254770:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762456.254772:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762456.254774:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504203008:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762456.894221:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762457.918219:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762458.942236:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762459.966219:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762460.990279:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762461.374168:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762461.374180:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762461.374186:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762461.374199:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2387 00000100:00080000:1.0:1789762461.374211:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762461.374220:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762461.374266:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504203136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762461.374301:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762461.374305:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762461.374308:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762461.374313:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2387 00000100:00080000:1.0:1789762461.374317:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762461.374321:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762461.374333:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762461.374349:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504203264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762461.374365:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504203136:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762461.374367:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762461.374383:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762461.374385:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762461.374391:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2387 00000100:00080000:1.0:1789762461.374394:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762461.374398:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762461.374404:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762461.374428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504203392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762461.374449:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762461.374453:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762461.374456:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762461.374459:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2387 00000100:00080000:1.0:1789762461.374464:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762461.374468:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762461.374474:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762461.374496:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504203520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762461.374501:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762461.374518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762461.374523:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762461.374526:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762461.374532:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2387 00000100:00080000:1.0:1789762461.374536:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762461.374541:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762461.374546:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762461.374559:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504203648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762461.374575:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762461.374579:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762461.374585:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762461.374587:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762461/real 1789762461] req@ffff9018c66a5c00 x1876699504203136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762461.374594:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2387 00000100:00080000:1.0:1789762461.374605:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762461.374618:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762461.374626:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762461.374635:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504203136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762461.374643:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504203776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762461.374675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762461.374685:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762461.374690:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504203136:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762461.374723:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504203264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762461.374763:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762461.374779:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762461/real 1789762461] req@ffff9018c66a6d80 x1876699504203264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762461.374805:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504203264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762461.374817:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762461.374821:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762461.374827:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504203264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762461.374839:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504203392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762461.374884:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762461.374915:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762461/real 1789762461] req@ffff9018c66a5f80 x1876699504203392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762461.374940:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504203392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762461.374952:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762461.374964:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762461.374970:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504203392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762461.374979:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504203520:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762461.375012:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762461.375026:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762461/real 1789762461] req@ffff9018c66a4000 x1876699504203520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762461.375140:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504203520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762461.375161:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762461.375166:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762461.375170:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504203520:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762461.375179:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504203648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762461.375210:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762461.375225:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762461/real 1789762461] req@ffff9018c66a7b80 x1876699504203648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762461.375238:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504203648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762461.375247:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762461.375251:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762461.375254:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504203648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762461.375260:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504203776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762461.375276:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762461.375286:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762461/real 1789762461] req@ffff9018c66a7800 x1876699504203776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762461.375299:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504203776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762461.375308:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762461.375311:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762461.375315:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504203776:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762462.014229:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762463.038255:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762464.062266:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762465.086246:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762466.110244:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762466.494210:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762466.494234:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762466.494239:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762466.494250:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2393 00000100:00080000:1.0:1789762466.494262:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762466.494270:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762466.494313:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504203904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762466.494348:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762466.494355:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762466.494358:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762466.494377:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2393 00000100:00080000:1.0:1789762466.494385:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762466.494392:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762466.494407:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762466.494434:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504203904:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762466.494436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504204032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762466.494462:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762466.494468:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762466.494470:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762466.494475:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2393 00000100:00080000:1.0:1789762466.494481:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762466.494486:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762466.494493:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762466.494508:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504204160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762466.494519:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762466.494525:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762466.494539:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762466.494542:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762466.494549:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2393 00000100:00080000:1.0:1789762466.494554:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762466.494558:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762466.494565:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762466.494578:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762466/real 1789762466] req@ffff9018c66a5500 x1876699504203904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762466.494586:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504204288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762466.494619:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762466.494626:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789762466.494627:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504203904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762466.494630:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762466.494644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762466.494648:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2393 00000100:00080000:0.0:1789762466.494655:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789762466.494656:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762466.494667:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762466.494672:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762466.494678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504203904:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762466.494688:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504204416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762466.494700:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762466.494704:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762466.494707:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762466.494715:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504204032:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789762466.494715:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2393 00000100:00080000:1.0:1789762466.494734:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762466.494741:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762466.494747:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762466.494754:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762466.494768:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504204544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762466.494777:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762466/real 1789762466] req@ffff9018c66a4e00 x1876699504204032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762466.494802:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504204032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762466.494813:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762466.494818:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762466.494823:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504204032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762466.494831:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504204160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762466.494855:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762466.494865:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762466/real 1789762466] req@ffff9018c66a6300 x1876699504204160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762466.494877:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504204160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762466.494886:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762466.494889:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762466.494892:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504204160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762466.494898:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504204288:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762466.494914:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762466.494923:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762466/real 1789762466] req@ffff9018c66a4700 x1876699504204288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762466.494935:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504204288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762466.494944:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762466.494946:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762466.494949:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504204288:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762466.494955:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504204416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762466.494970:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762466.494979:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762466/real 1789762466] req@ffff9018c66a6680 x1876699504204416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762466.494998:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504204416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762466.495006:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762466.495009:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762466.495012:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504204416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762466.495031:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504204544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762466.495163:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762466.495178:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762466/real 1789762466] req@ffff9018c66a5180 x1876699504204544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762466.495191:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504204544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762466.495200:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762466.495203:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762466.495207:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504204544:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762467.134297:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762468.158214:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762469.182191:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762470.206226:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762471.230181:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762471.614110:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762471.614118:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762471.614122:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762471.614130:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2398 00000100:00080000:1.0:1789762471.614139:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762471.614145:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762471.614175:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504204672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762471.614196:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762471.614198:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762471.614200:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762471.614203:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2398 00000100:00080000:1.0:1789762471.614206:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762471.614210:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762471.614218:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762471.614238:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504204672:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762471.614238:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504204800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762471.614252:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762471.614255:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762471.614257:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762471.614260:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2398 00000100:00080000:1.0:1789762471.614263:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762471.614265:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762471.614268:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762471.614277:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504204928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762471.614284:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762471.614287:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762471.614288:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762471.614290:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2398 00000100:00080000:1.0:1789762471.614293:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762471.614298:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762471.614298:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762471.614306:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762471.614316:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504205056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762471.614334:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762471.614337:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762471/real 1789762471] req@ffff9018c66a5880 x1876699504204672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762471.614338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762471.614341:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762471.614349:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2398 00000100:00080000:1.0:1789762471.614356:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762471.614363:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504204672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762471.614364:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762471.614373:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789762471.614379:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762471.614383:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762471.614387:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504204672:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762471.614401:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504205184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762471.614410:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504204800:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789762471.614418:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762471.614424:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762471.614426:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762471.614431:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2398 00000800:00000100:0.0:1789762471.614434:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762471.614438:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762471.614444:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762471.614449:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762471/real 1789762471] req@ffff9018c66a4380 x1876699504204800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762471.614452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762471.614465:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504204800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762471.614472:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789762471.614474:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504205312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762471.614478:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762471.614485:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504204800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762471.614496:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504204928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762471.614512:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762471.614520:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762471/real 1789762471] req@ffff9018c66a6a00 x1876699504204928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762471.614528:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504204928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762471.614534:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762471.614537:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762471.614539:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504204928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762471.614543:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504205056:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762471.614554:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762471.614560:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762471/real 1789762471] req@ffff9018c66a7480 x1876699504205056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762471.614568:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504205056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762471.614573:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762471.614575:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762471.614577:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504205056:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762471.614587:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504205184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762471.614598:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762471.614604:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762471/real 1789762471] req@ffff9018c66a4a80 x1876699504205184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762471.614611:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504205184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762471.614617:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762471.614618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762471.614620:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504205184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762471.614624:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504205312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762471.614633:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762471.614639:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762471/real 1789762471] req@ffff9018c66a7100 x1876699504205312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762471.614647:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504205312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762471.614652:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762471.614653:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762471.614655:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504205312:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762472.254184:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762473.278244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762474.302295:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762475.326194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762476.350226:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762476.734140:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762476.734150:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762476.734155:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762476.734165:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2403 00000100:00080000:0.0:1789762476.734176:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762476.734184:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762476.734227:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504205440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762476.734253:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762476.734256:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762476.734258:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762476.734261:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2403 00000100:00080000:0.0:1789762476.734263:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762476.734266:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762476.734276:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762476.734290:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504205440:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762476.734292:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504205568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762476.734307:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762476.734311:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762476.734313:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762476.734318:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2403 00000100:00080000:0.0:1789762476.734322:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762476.734327:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762476.734333:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762476.734346:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504205696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762476.734354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762476.734357:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762476.734358:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762476.734361:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2403 00000100:00080000:0.0:1789762476.734364:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762476.734368:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762476.734369:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762476.734377:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762476.734386:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504205824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762476.734393:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762476.734396:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762476.734397:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762476.734400:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2403 00000100:00080000:0.0:1789762476.734403:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762476.734406:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762476.734409:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762476.734416:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762476/real 1789762476] req@ffff9018c8b9b100 x1876699504205440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762476.734419:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504205952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762476.734445:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762476.734452:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762476.734456:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762476.734459:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2403 00000100:00080000:0.0:1789762476.734462:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762476.734464:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762476.734469:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504205440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789762476.734470:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762476.734482:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504206080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762476.734484:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762476.734493:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762476.734498:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504205440:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762476.734522:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504205568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762476.734539:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762476.734548:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762476/real 1789762476] req@ffff9018c8b9b480 x1876699504205568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762476.734556:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504205568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762476.734562:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762476.734564:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762476.734566:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504205568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762476.734569:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504205696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762476.734580:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762476.734586:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762476/real 1789762476] req@ffff9018c8b98000 x1876699504205696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762476.734593:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504205696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762476.734602:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762476.734604:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762476.734606:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504205696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762476.734610:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504205824:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762476.734620:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762476.734626:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762476/real 1789762476] req@ffff9018c8b9aa00 x1876699504205824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762476.734633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504205824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762476.734638:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762476.734640:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762476.734642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504205824:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762476.734645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504205952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762476.734655:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762476.734661:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762476/real 1789762476] req@ffff9018c8b98e00 x1876699504205952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762476.734668:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504205952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762476.734673:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762476.734674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762476.734677:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504205952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762476.734681:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504206080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762476.734692:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762476.734698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762476/real 1789762476] req@ffff9018c8b9a680 x1876699504206080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762476.734704:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504206080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762476.734709:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762476.734711:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762476.734713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504206080:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762477.374217:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762478.398248:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762479.422223:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762480.446236:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762481.470157:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762481.854147:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762481.854161:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762481.854166:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762481.854179:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2408 00000100:00080000:0.0:1789762481.854193:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762481.854203:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762481.854258:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504206208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762481.854299:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762481.854302:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762481.854304:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762481.854307:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2408 00000100:00080000:0.0:1789762481.854310:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762481.854314:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762481.854329:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762481.854342:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504206336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762481.854350:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762481.854353:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762481.854354:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762481.854358:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2408 00000100:00080000:0.0:1789762481.854362:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762481.854368:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762481.854374:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762481.854388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504206464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762481.854402:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762481.854405:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504206208:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762481.854409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762481.854419:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762481.854428:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2408 00000100:00080000:0.0:1789762481.854434:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762481.854441:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762481.854448:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762481.854464:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504206592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762481.854476:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762481.854479:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762481.854481:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789762481.854484:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762481.854488:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2408 00000100:00080000:0.0:1789762481.854493:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762481.854499:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762481.854507:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762481.854524:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504206720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762481.854529:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762481/real 1789762481] req@ffff9018c8b98380 x1876699504206208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762481.854568:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762481.854572:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762481.854574:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762481.854583:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2408 00000100:00080000:0.0:1789762481.854587:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762481.854590:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762481.854592:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504206208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789762481.854596:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762481.854618:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789762481.854621:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504206848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762481.854626:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762481.854643:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504206208:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762481.854669:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504206336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762481.854696:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762481.854710:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762481/real 1789762481] req@ffff9018c8b99c00 x1876699504206336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762481.854720:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504206336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762481.854727:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762481.854729:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762481.854732:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504206336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762481.854752:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504206464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762481.854764:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762481.854771:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762481/real 1789762481] req@ffff9018c8b99500 x1876699504206464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762481.854778:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504206464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762481.854784:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762481.854785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762481.854787:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504206464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762481.854791:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504206592:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762481.854801:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762481.854807:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762481/real 1789762481] req@ffff9018c8b9ad80 x1876699504206592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762481.854815:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504206592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762481.854821:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762481.854823:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762481.854825:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504206592:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762481.854829:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504206720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762481.854839:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762481.854844:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762481/real 1789762481] req@ffff9018c8b9a300 x1876699504206720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762481.854851:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504206720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762481.854856:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762481.854858:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762481.854860:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504206720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762481.854864:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504206848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762481.854873:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762481.854879:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762481/real 1789762481] req@ffff9018c8b9a680 x1876699504206848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762481.854885:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504206848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762481.854890:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762481.854892:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762481.854895:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504206848:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762482.494184:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762483.518180:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762484.542156:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762485.566195:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762486.590157:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762486.974171:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762486.974181:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762486.974184:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762486.974194:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2413 00000100:00080000:1.0:1789762486.974204:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762486.974209:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762486.974256:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504206976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762486.974285:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762486.974288:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762486.974290:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762486.974293:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2413 00000100:00080000:1.0:1789762486.974296:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762486.974300:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762486.974308:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762486.974331:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504207104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762486.974341:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762486.974345:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762486.974350:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762486.974352:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504206976:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762486.974358:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2413 00000100:00080000:1.0:1789762486.974369:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762486.974374:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762486.974378:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762486.974387:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504207232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762486.974393:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762486.974396:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762486.974397:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762486.974400:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2413 00000100:00080000:1.0:1789762486.974404:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762486.974407:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762486.974411:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762486.974421:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504207360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762486.974426:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762486.974430:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762486.974431:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762486.974435:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2413 00000100:00080000:1.0:1789762486.974438:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762486.974441:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762486.974444:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762486.974449:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762486.974454:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504207488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762486.974462:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762486.974465:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762486.974467:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762486.974471:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2413 00000100:00080000:1.0:1789762486.974474:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762486.974477:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762486.974481:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762486.974490:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504207616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762486.974524:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762486/real 1789762486] req@ffff9018c66a7100 x1876699504206976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762486.974543:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504206976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762486.974554:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762486.974558:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762486.974562:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504206976:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762486.974598:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504207104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762486.974615:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762486.974624:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762486/real 1789762486] req@ffff9018c66a4a80 x1876699504207104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762486.974631:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504207104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762486.974637:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762486.974639:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762486.974642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504207104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762486.974645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504207232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762486.974657:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762486.974663:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762486/real 1789762486] req@ffff9018c66a7480 x1876699504207232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762486.974670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504207232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762486.974675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762486.974677:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762486.974679:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504207232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762486.974683:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504207360:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762486.974694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762486.974700:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762486/real 1789762486] req@ffff9018c66a6a00 x1876699504207360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762486.974707:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504207360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762486.974713:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762486.974714:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762486.974716:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504207360:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762486.974720:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504207488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762486.974729:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762486.974735:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762486/real 1789762486] req@ffff9018c66a4380 x1876699504207488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762486.974746:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504207488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762486.974751:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762486.974763:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762486.974780:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504207488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762486.974784:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504207616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762486.974796:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762486.974802:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762486/real 1789762486] req@ffff9018c66a5880 x1876699504207616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762486.974809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504207616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762486.974814:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762486.974816:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762486.974818:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504207616:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762487.614211:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762488.638151:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762489.662230:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762490.686178:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762491.710205:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762492.094105:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762492.094127:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762492.094132:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762492.094145:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2418 00000100:00080000:1.0:1789762492.094159:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762492.094166:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762492.094208:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504207744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762492.094236:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762492.094240:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762492.094241:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762492.094245:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2418 00000100:00080000:1.0:1789762492.094249:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762492.094253:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762492.094263:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762492.094281:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504207744:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762492.094282:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504207872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762492.094309:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762492.094311:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762492.094314:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762492.094319:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2418 00000100:00080000:1.0:1789762492.094322:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762492.094326:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762492.094331:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762492.094343:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504208000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762492.094351:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762492.094353:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762492.094354:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762492.094358:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2418 00000100:00080000:1.0:1789762492.094361:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762492.094364:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762492.094368:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762492.094378:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504208128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762492.094387:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762492.094389:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762492.094391:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762492.094394:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2418 00000100:00080000:1.0:1789762492.094397:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762492.094402:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762492.094406:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762492.094414:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504208256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762492.094422:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762492.094423:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504207872:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789762492.094435:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762492.094438:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762492.094444:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2418 00000100:00080000:1.0:1789762492.094448:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762492.094452:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762492.094457:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762492.094467:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504208000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762492.094472:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504208384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762492.094494:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504208128:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762492.094519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504208256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762492.094544:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504208384:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762492.734159:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762493.758222:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762494.782183:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762495.166281:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789762495.166318:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762495.166334:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762495.166373:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762495.166382:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762495.166393:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762495.166404:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762495.166416:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789762495.166428:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762492/real 1789762495] req@ffff9018c66a4e00 x1876699504208256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762495.166466:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504208256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762495.166480:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762495.166485:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762495.166490:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504208256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762495.166499:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762492/real 1789762495] req@ffff9018c66a5500 x1876699504208384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762495.166513:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504208384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762495.166523:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762495.166527:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762495.166531:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504208384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762495.166539:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762492/real 1789762495] req@ffff9018c66a6680 x1876699504207872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762495.166553:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504207872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762495.166564:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762495.166567:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762495.166579:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504207872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762495.166587:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762492/real 1789762495] req@ffff9018c66a4700 x1876699504208000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762495.166601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504208000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762495.166610:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762495.166614:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762495.166618:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504208000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762495.166626:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762492/real 1789762495] req@ffff9018c66a6300 x1876699504208128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762495.166639:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504208128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762495.166649:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762495.166653:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762495.166656:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504208128:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762495.166662:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762492/real 1789762495] req@ffff9018c66a5180 x1876699504207744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762495.166676:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504207744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762495.166685:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762495.166688:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762495.166692:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504207744:192.168.203.158@tcp:250:kworker.0 00010000:00010000:3.0:1789762495.806181:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762496.830193:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762497.214190:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762497.214197:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762497.214201:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762497.214210:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2423 00000100:00080000:2.0:1789762497.214219:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762497.214225:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762497.214255:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504208512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762497.214283:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762497.214286:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762497.214288:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762497.214292:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2423 00000100:00080000:2.0:1789762497.214297:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762497.214301:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762497.214312:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762497.214325:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504208512:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762497.214330:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504208640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762497.214346:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762497.214349:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762497.214351:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762497.214356:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2423 00000100:00080000:2.0:1789762497.214360:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762497.214365:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762497.214370:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762497.214385:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504208768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762497.214396:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762497.214400:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762497.214402:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762497.214406:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762497.214412:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2423 00000100:00080000:2.0:1789762497.214417:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762497.214421:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762497.214425:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762497.214435:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504208896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762497.214443:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762497.214445:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762497.214447:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762497.214451:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2423 00000100:00100000:0.0:1789762497.214452:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762497/real 1789762497] req@ffff9018d0f26d80 x1876699504208512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762497.214454:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762497.214461:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762497.214471:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762497.214483:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504208512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762497.214487:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504209024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762497.214512:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762497.214512:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762497.214520:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762497.214524:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789762497.214525:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762497.214534:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504208512:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762497.214534:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2423 00000100:00080000:2.0:1789762497.214541:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762497.214553:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762497.214559:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762497.214567:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504208640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762497.214574:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504209152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762497.214603:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762497.214621:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762497/real 1789762497] req@ffff9018d0f25f80 x1876699504208640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762497.214634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504208640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762497.214642:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762497.214646:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762497.214649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504208640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762497.214656:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504208768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762497.214676:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762497.214686:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762497/real 1789762497] req@ffff9018d0f27100 x1876699504208768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762497.214697:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504208768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762497.214707:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762497.214711:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762497.214714:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504208768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762497.214721:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504208896:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762497.214743:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762497.214753:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762497/real 1789762497] req@ffff9018d0f26300 x1876699504208896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762497.214765:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504208896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762497.214773:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762497.214777:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762497.214780:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504208896:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762497.214786:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504209024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762497.214807:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762497.214818:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762497/real 1789762497] req@ffff9018d0f25c00 x1876699504209024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762497.214848:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504209024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762497.214856:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762497.214860:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762497.214863:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504209024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762497.214882:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504209152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762497.214903:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762497.214915:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762497/real 1789762497] req@ffff9018d0f26a00 x1876699504209152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762497.214928:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504209152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762497.214936:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762497.214940:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762497.214943:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504209152:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762497.854203:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762498.878198:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762499.902257:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762500.926268:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762501.950184:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762502.334106:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762502.334116:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762502.334121:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762502.334131:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2428 00000100:00080000:2.0:1789762502.334143:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762502.334151:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762502.334194:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504209280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762502.334220:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762502.334223:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762502.334224:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762502.334228:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2428 00000100:00080000:2.0:1789762502.334230:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762502.334233:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762502.334243:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762502.334254:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504209408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762502.334262:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762502.334264:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762502.334267:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762502.334270:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2428 00000100:00080000:2.0:1789762502.334273:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762502.334275:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762502.334277:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762502.334284:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504209536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762502.334293:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762502.334297:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762502.334303:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762502.334305:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504209280:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762502.334311:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2428 00000100:00080000:2.0:1789762502.334320:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762502.334327:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762502.334334:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762502.334350:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504209664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762502.334362:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762502.334365:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762502.334366:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762502.334370:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2428 00000100:00080000:2.0:1789762502.334382:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762502.334389:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789762502.334392:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789762502.334397:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762502.334411:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504209792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762502.334418:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762502.334421:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762502.334422:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762502.334425:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2428 00000100:00080000:2.0:1789762502.334427:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762502.334429:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762502.334432:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762502.334439:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504209920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762502.334452:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762502/real 1789762502] req@ffff9018d0f24380 x1876699504209280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762502.334472:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504209280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762502.334485:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762502.334489:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762502.334494:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504209280:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762502.334502:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504209408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762502.334527:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762502.334539:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762502/real 1789762502] req@ffff9018d0f27800 x1876699504209408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762502.334551:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504209408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762502.334560:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762502.334563:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762502.334567:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504209408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762502.334593:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504209536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762502.334612:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762502.334623:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762502/real 1789762502] req@ffff9018d0f25180 x1876699504209536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762502.334635:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504209536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762502.334643:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762502.334647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762502.334651:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504209536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762502.334658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504209664:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762502.334675:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762502.334685:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762502/real 1789762502] req@ffff9018d0f24a80 x1876699504209664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762502.334696:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504209664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762502.334704:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762502.334708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762502.334711:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504209664:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762502.334718:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504209792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762502.334735:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762502.334745:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762502/real 1789762502] req@ffff9018d0f24000 x1876699504209792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762502.334756:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504209792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762502.334764:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762502.334767:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762502.334771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504209792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762502.334777:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504209920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762502.334809:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762502.334821:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762502/real 1789762502] req@ffff9018d0f24e00 x1876699504209920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762502.334841:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504209920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762502.334849:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762502.334853:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762502.334857:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504209920:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762502.974232:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762503.998179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762505.022234:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762506.046195:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762507.070255:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762507.454189:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762507.454203:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762507.454208:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762507.454220:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2434 00000100:00080000:1.0:1789762507.454233:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762507.454240:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762507.454283:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504210048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762507.454316:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762507.454320:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762507.454321:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762507.454325:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2434 00000100:00080000:1.0:1789762507.454327:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762507.454330:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762507.454340:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762507.454352:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504210176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762507.454359:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762507.454362:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762507.454364:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762507.454367:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2434 00000100:00080000:1.0:1789762507.454369:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762507.454371:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762507.454376:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762507.454387:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504210048:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762507.454388:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504210304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762507.454412:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762507.454416:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762507.454418:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762507.454424:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2434 00000100:00080000:1.0:1789762507.454429:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762507.454434:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762507.454441:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762507.454454:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504210432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762507.454462:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762507.454465:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762507.454467:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762507.454469:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2434 00000100:00080000:1.0:1789762507.454473:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762507.454482:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762507.454485:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762507.454491:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762507.454503:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504210560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762507.454511:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762507.454513:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762507.454514:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762507.454518:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2434 00000100:00080000:1.0:1789762507.454522:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762507.454526:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762507.454529:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762507.454543:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504210688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762507.454544:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762507/real 1789762507] req@ffff9018c66a7800 x1876699504210048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762507.454587:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504210048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762507.454601:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762507.454608:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762507.454613:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504210048:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762507.454622:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504210176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762507.454647:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762507.454660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762507/real 1789762507] req@ffff9018c66a7b80 x1876699504210176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762507.454673:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504210176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762507.454683:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762507.454687:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762507.454691:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504210176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762507.454718:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504210304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762507.454738:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762507.454749:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762507/real 1789762507] req@ffff9018c66a4000 x1876699504210304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762507.454777:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504210304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762507.454785:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762507.454789:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762507.454793:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504210304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762507.454800:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504210432:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762507.454821:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762507.454833:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762507/real 1789762507] req@ffff9018c66a5f80 x1876699504210432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762507.454845:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504210432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762507.454854:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762507.454857:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762507.454861:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504210432:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762507.454867:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504210560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762507.454885:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762507.454895:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762507/real 1789762507] req@ffff9018c66a6d80 x1876699504210560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762507.454906:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504210560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762507.454914:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762507.454917:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762507.454921:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504210560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762507.454927:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504210688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762507.454943:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762507.454953:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762507/real 1789762507] req@ffff9018c66a5c00 x1876699504210688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762507.454964:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504210688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762507.454973:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762507.454976:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762507.454980:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504210688:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762508.094182:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762509.118201:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762510.142178:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762511.166235:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762512.190194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762512.574166:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762512.574176:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762512.574180:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762512.574192:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2439 00000100:00080000:0.0:1789762512.574204:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762512.574211:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762512.574253:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504210816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762512.574280:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762512.574283:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762512.574285:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762512.574289:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2439 00000100:00080000:0.0:1789762512.574293:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762512.574297:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762512.574310:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762512.574325:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504210944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762512.574330:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504210816:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762512.574344:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762512.574348:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762512.574349:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762512.574353:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2439 00000100:00080000:0.0:1789762512.574356:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762512.574359:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762512.574364:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762512.574377:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504211072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762512.574388:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762512.574392:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762512.574394:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762512.574398:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2439 00000100:00080000:0.0:1789762512.574403:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762512.574407:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762512.574413:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789762512.574420:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762512.574428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504211200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762512.574436:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762512.574439:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762512.574441:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762512.574444:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2439 00000100:00080000:0.0:1789762512.574447:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762512.574451:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762512.574455:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762512.574462:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504211328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762512.574468:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762512.574471:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762512.574472:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762512.574478:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2439 00000100:00100000:2.0:1789762512.574480:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762512/real 1789762512] req@ffff9018c8b98e00 x1876699504210816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762512.574481:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762512.574488:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762512.574495:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762512.574505:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504211456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762512.574510:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504210816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762512.574523:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762512.574527:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762512.574531:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504210816:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762512.574557:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504210944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762512.574576:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762512.574586:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762512/real 1789762512] req@ffff9018c8b9aa00 x1876699504210944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762512.574594:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504210944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762512.574600:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762512.574603:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762512.574605:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504210944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762512.574609:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504211072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762512.574620:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762512.574626:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762512/real 1789762512] req@ffff9018c8b98000 x1876699504211072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762512.574633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504211072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762512.574639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762512.574641:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762512.574642:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504211072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762512.574647:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504211200:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762512.574657:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762512.574663:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762512/real 1789762512] req@ffff9018c8b9b480 x1876699504211200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762512.574670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504211200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762512.574676:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762512.574678:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762512.574680:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504211200:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762512.574683:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504211328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762512.574693:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762512.574698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762512/real 1789762512] req@ffff9018c8b9b100 x1876699504211328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762512.574705:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504211328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762512.574731:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762512.574734:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762512.574737:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504211328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762512.574741:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504211456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762512.574758:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762512.574765:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762512/real 1789762512] req@ffff9018c8b9b800 x1876699504211456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762512.574774:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504211456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762512.574779:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762512.574781:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762512.574783:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504211456:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762513.214216:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762514.238307:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762515.262234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762516.286256:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762517.310251:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762517.694149:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762517.694160:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762517.694165:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762517.694175:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2444 00000100:00080000:0.0:1789762517.694185:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762517.694193:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762517.694229:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504211584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762517.694258:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762517.694262:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762517.694264:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762517.694268:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2444 00000100:00080000:0.0:1789762517.694273:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762517.694279:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762517.694289:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762517.694303:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504211584:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762517.694306:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504211712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762517.694328:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762517.694332:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762517.694335:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762517.694339:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2444 00000100:00080000:0.0:1789762517.694343:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762517.694347:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762517.694352:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762517.694358:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762517.694367:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504211840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762517.694378:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762517.694382:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762517.694384:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762517.694389:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2444 00000100:00080000:0.0:1789762517.694393:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762517.694394:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762517/real 1789762517] req@ffff9018c8b9bb80 x1876699504211584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762517.694401:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762517.694413:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762517.694417:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504211584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762517.694427:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789762517.694428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504211968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762517.694433:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762517.694440:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504211584:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762517.694453:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762517.694457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762517.694459:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762517.694464:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2444 00000100:00080000:0.0:1789762517.694468:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762517.694472:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762517.694476:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504211712:192.168.203.158@tcp:38:kworker.0 00000008:00100000:0.0:1789762517.694479:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762517.694498:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504212096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762517.694505:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762517.694515:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762517.694519:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762517/real 1789762517] req@ffff9018c8b99880 x1876699504211712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762517.694521:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762517.694525:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762517.694536:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2444 00000100:00100000:2.0:1789762517.694541:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504211712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762517.694542:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762517.694551:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762517.694561:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762517.694563:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762517.694569:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762517.694572:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504211712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762517.694578:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504212224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762517.694579:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504211840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762517.694600:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762517.694609:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762517/real 1789762517] req@ffff9018c8b98a80 x1876699504211840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762517.694617:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504211840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762517.694623:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762517.694625:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762517.694627:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504211840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762517.694631:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504211968:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762517.694648:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762517.694656:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762517/real 1789762517] req@ffff9018c8b99f80 x1876699504211968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762517.694666:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504211968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762517.694672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762517.694674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762517.694691:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504211968:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762517.694706:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504212096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762517.694720:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762517.694727:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762517/real 1789762517] req@ffff9018c8b98700 x1876699504212096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762517.694735:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504212096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762517.694740:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762517.694742:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762517.694744:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504212096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762517.694748:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504212224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762517.694757:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762517.694763:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762517/real 1789762517] req@ffff9018c8b9bb80 x1876699504212224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762517.694770:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504212224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762517.694775:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762517.694776:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762517.694778:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504212224:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762518.334181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762519.358144:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762520.382229:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762521.406144:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762522.430227:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762522.814145:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762522.814156:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762522.814160:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762522.814171:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2449 00000100:00080000:0.0:1789762522.814184:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762522.814191:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762522.814240:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504212352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762522.814270:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762522.814274:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762522.814275:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762522.814278:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2449 00000100:00080000:0.0:1789762522.814280:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762522.814283:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762522.814292:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762522.814302:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504212480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762522.814309:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762522.814311:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762522.814312:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762522.814314:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2449 00000100:00080000:0.0:1789762522.814317:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762522.814319:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762522.814322:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762522.814331:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504212608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762522.814337:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762522.814340:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762522.814341:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762522.814344:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2449 00000100:00080000:0.0:1789762522.814346:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762522.814351:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762522.814360:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762522.814361:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504212352:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762522.814372:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504212736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762522.814383:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762522.814386:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762522.814387:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762522.814390:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2449 00000100:00080000:0.0:1789762522.814392:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762522.814395:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762522.814398:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762522.814407:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504212864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762522.814412:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762522.814415:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762522.814417:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762522.814420:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2449 00000100:00080000:0.0:1789762522.814423:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762522.814425:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762522.814428:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762522.814431:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762522.814441:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504212992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762522.814476:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762522/real 1789762522] req@ffff9018c8b9b800 x1876699504212352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762522.814497:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504212352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762522.814509:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762522.814514:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762522.814519:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504212352:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762522.814528:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504212480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762522.814550:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762522.814560:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762522/real 1789762522] req@ffff9018c8b9b100 x1876699504212480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762522.814568:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504212480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762522.814574:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762522.814576:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762522.814578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504212480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762522.814582:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504212608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762522.814593:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762522.814598:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762522/real 1789762522] req@ffff9018c8b9b480 x1876699504212608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762522.814606:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504212608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762522.814611:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762522.814613:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762522.814615:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504212608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762522.814634:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504212736:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762522.814667:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762522.814675:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762522/real 1789762522] req@ffff9018c8b98000 x1876699504212736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762522.814684:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504212736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762522.814689:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762522.814691:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762522.814693:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504212736:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762522.814697:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504212864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762522.814707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762522.814713:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762522/real 1789762522] req@ffff9018c8b9aa00 x1876699504212864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762522.814720:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504212864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762522.814730:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762522.814732:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762522.814734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504212864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762522.814738:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504212992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762522.814747:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762522.814752:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762522/real 1789762522] req@ffff9018c8b98e00 x1876699504212992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762522.814759:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504212992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762522.814764:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762522.814766:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762522.814768:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504212992:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762523.454198:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762524.478210:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762525.502186:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762526.526218:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762527.550169:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762527.934137:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762527.934146:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762527.934149:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762527.934157:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2454 00000100:00080000:0.0:1789762527.934166:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762527.934173:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762527.934204:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504213120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762527.934226:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762527.934231:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762527.934233:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762527.934237:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2454 00000100:00080000:0.0:1789762527.934241:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762527.934246:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762527.934257:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762527.934275:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504213248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762527.934285:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504213120:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762527.934299:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762527.934303:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762527.934305:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762527.934309:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2454 00000100:00080000:0.0:1789762527.934312:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762527.934315:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762527.934321:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762527.934336:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504213376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762527.934353:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762527.934358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762527.934360:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762527.934365:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2454 00000100:00080000:0.0:1789762527.934371:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762527.934378:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762527.934383:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762527.934390:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762527.934404:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504213504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762527.934413:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762527.934415:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762527.934417:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762527.934420:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2454 00000100:00080000:0.0:1789762527.934423:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762527.934427:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762527.934433:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762527.934442:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762527/real 1789762527] req@ffff9018c8b9a680 x1876699504213120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762527.934444:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504213632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762527.934514:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762527.934518:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762527.934519:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762527.934524:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2454 00000100:00100000:2.0:1789762527.934525:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504213120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762527.934531:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762527.934543:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762527.934555:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762527.934572:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504213760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762527.934575:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762527.934595:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762527.934616:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504213120:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762527.934646:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504213248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762527.934670:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762527.934685:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762527/real 1789762527] req@ffff9018c8b9a300 x1876699504213248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762527.934699:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504213248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762527.934710:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762527.934714:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762527.934718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504213248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762527.934724:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504213376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762527.934741:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762527.934750:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762527/real 1789762527] req@ffff9018c8b9ad80 x1876699504213376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762527.934758:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504213376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762527.934772:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762527.934778:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762527.934780:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504213376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762527.934786:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504213504:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762527.934800:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762527.934807:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762527/real 1789762527] req@ffff9018c8b99500 x1876699504213504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762527.934815:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504213504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762527.934821:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762527.934823:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762527.934826:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504213504:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762527.934829:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504213632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762527.934840:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762527.934845:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762527/real 1789762527] req@ffff9018c8b99c00 x1876699504213632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762527.934853:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504213632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762527.934858:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762527.934860:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762527.934862:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504213632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762527.934866:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504213760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762527.934880:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762527.934891:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762527/real 1789762527] req@ffff9018c8b98380 x1876699504213760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762527.934904:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504213760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762527.934914:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762527.934917:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762527.934921:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504213760:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762528.574233:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762529.598150:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762530.622180:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762531.646157:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762532.670173:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762533.054147:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762533.054161:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762533.054168:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762533.054182:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2459 00000100:00080000:0.0:1789762533.054197:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762533.054205:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762533.054244:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504213888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762533.054274:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762533.054278:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762533.054279:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762533.054283:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2459 00000100:00080000:0.0:1789762533.054288:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762533.054294:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762533.054306:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762533.054324:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504214016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762533.054325:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504213888:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762533.054354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762533.054358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762533.054360:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762533.054364:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2459 00000100:00080000:0.0:1789762533.054370:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762533.054373:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762533.054377:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762533.054389:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504214144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762533.054397:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762533.054400:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762533.054402:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762533.054405:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2459 00000100:00080000:0.0:1789762533.054408:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762533.054411:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789762533.054418:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789762533.054422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762533.054434:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504214272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762533.054441:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762533.054444:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762533.054447:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762533.054453:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2459 00000100:00080000:0.0:1789762533.054456:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762533.054460:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762533/real 1789762533] req@ffff9018c8b98380 x1876699504213888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762533.054461:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762533.054471:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762533.054487:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504214400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762533.054495:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504213888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762533.054521:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762533.054535:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762533.054537:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762533.054539:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762533.054547:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2459 00000100:00080000:1.0:1789762533.054554:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789762533.054557:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762533.054575:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504213888:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762533.054582:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762533.054591:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762533.054608:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504214528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762533.054609:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504214016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762533.054651:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762533.054665:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762533/real 1789762533] req@ffff9018c8b99c00 x1876699504214016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762533.054676:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504214016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762533.054683:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762533.054686:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762533.054689:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504214016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762533.054694:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504214144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762533.054709:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762533.054724:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762533/real 1789762533] req@ffff9018c8b99500 x1876699504214144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762533.054733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504214144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762533.054744:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762533.054747:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762533.054749:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504214144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762533.054754:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504214272:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762533.054773:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762533.054801:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762533/real 1789762533] req@ffff9018c8b9ad80 x1876699504214272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762533.054824:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504214272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762533.054831:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762533.054834:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762533.054838:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504214272:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762533.054843:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504214400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762533.054862:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762533.054869:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762533/real 1789762533] req@ffff9018c8b9a300 x1876699504214400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762533.054877:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504214400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762533.054883:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762533.054885:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762533.054888:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504214400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762533.054900:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504214528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762533.054912:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762533.054919:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762533/real 1789762533] req@ffff9018c8b9a680 x1876699504214528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762533.054926:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504214528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762533.054931:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762533.054933:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762533.054935:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504214528:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762533.694188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762534.718202:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762535.742164:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762536.766203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762537.790191:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762538.174089:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762538.174101:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762538.174106:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762538.174118:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2464 00000100:00080000:0.0:1789762538.174133:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762538.174142:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762538.174190:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504214656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762538.174215:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762538.174218:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762538.174220:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762538.174223:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2464 00000100:00080000:0.0:1789762538.174226:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762538.174230:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762538.174239:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762538.174250:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504214784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762538.174258:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762538.174260:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762538.174262:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762538.174264:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2464 00000100:00080000:0.0:1789762538.174266:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762538.174269:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762538.174272:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762538.174276:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504214656:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762538.174283:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504214912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762538.174296:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762538.174299:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762538.174301:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762538.174306:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2464 00000100:00080000:0.0:1789762538.174310:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762538.174315:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762538.174321:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762538.174334:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504215040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762538.174347:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762538.174350:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762538.174352:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762538.174354:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2464 00000100:00080000:0.0:1789762538.174357:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762538.174360:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762538.174363:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762538.174368:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762538.174375:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504215168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762538.174383:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762538.174386:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762538.174388:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762538.174390:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2464 00000100:00080000:0.0:1789762538.174393:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762538.174396:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762538.174399:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762538.174405:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504215296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762538.174425:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762538/real 1789762538] req@ffff9018c8b98e00 x1876699504214656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762538.174448:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504214656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762538.174461:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762538.174466:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762538.174471:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504214656:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762538.174479:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504214784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762538.174506:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762538.174538:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762538/real 1789762538] req@ffff9018c8b9aa00 x1876699504214784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762538.174553:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504214784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762538.174562:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762538.174566:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762538.174570:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504214784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762538.174599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504214912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762538.174621:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762538.174634:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762538/real 1789762538] req@ffff9018c8b98000 x1876699504214912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762538.174646:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504214912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762538.174655:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762538.174658:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762538.174661:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504214912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762538.174667:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504215040:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762538.174688:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762538.174700:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762538/real 1789762538] req@ffff9018c8b9b480 x1876699504215040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762538.174711:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504215040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762538.174720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762538.174723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762538.174726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504215040:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762538.174732:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504215168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762538.174752:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762538.174764:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762538/real 1789762538] req@ffff9018c8b9b100 x1876699504215168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762538.174776:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504215168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762538.174784:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762538.174787:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762538.174791:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504215168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762538.174797:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504215296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762538.174817:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762538.174828:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762538/real 1789762538] req@ffff9018c8b9b800 x1876699504215296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762538.174840:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504215296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762538.174849:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762538.174852:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762538.174856:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504215296:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762538.814230:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762539.838227:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762540.862222:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762541.886170:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762542.910197:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762543.294123:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762543.294136:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762543.294141:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762543.294155:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2469 00000100:00080000:0.0:1789762543.294178:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762543.294186:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762543.294230:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504215424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762543.294253:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762543.294256:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762543.294257:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762543.294260:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2469 00000100:00080000:0.0:1789762543.294263:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762543.294267:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762543.294276:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762543.294286:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504215424:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762543.294288:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504215552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762543.294302:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762543.294305:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762543.294306:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762543.294308:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2469 00000100:00080000:0.0:1789762543.294310:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762543.294312:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762543.294315:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762543.294324:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504215680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762543.294330:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762543.294333:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762543.294335:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762543.294337:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2469 00000100:00080000:0.0:1789762543.294340:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762543.294342:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762543.294346:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762543.294353:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504215808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762543.294359:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762543.294362:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762543.294363:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762543.294366:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2469 00000100:00080000:0.0:1789762543.294368:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762543.294371:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762543.294374:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762543.294376:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762543.294384:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504215936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762543.294390:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762543.294393:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762543.294395:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762543.294398:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2469 00000100:00080000:0.0:1789762543.294401:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762543.294403:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762543.294406:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762543.294415:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504216064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762543.294423:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762543/real 1789762543] req@ffff9018c8b9bb80 x1876699504215424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762543.294442:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504215424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762543.294451:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762543.294454:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762543.294457:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504215424:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762543.294500:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504215552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762543.294520:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762543.294530:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762543/real 1789762543] req@ffff9018c8b98700 x1876699504215552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762543.294538:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504215552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762543.294544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762543.294546:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762543.294549:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504215552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762543.294553:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504215680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762543.294564:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762543.294570:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762543/real 1789762543] req@ffff9018c8b99f80 x1876699504215680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762543.294578:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504215680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762543.294583:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762543.294584:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762543.294586:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504215680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762543.294590:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504215808:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762543.294601:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762543.294607:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762543/real 1789762543] req@ffff9018c8b98a80 x1876699504215808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762543.294619:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504215808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762543.294627:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762543.294630:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762543.294634:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504215808:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762543.294640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504215936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762543.294662:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762543.294672:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762543/real 1789762543] req@ffff9018c8b99880 x1876699504215936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762543.294680:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504215936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762543.294694:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762543.294697:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762543.294699:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504215936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762543.294703:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504216064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762543.294717:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762543.294725:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762543/real 1789762543] req@ffff9018c8b9b800 x1876699504216064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762543.294732:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504216064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762543.294737:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762543.294739:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762543.294741:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504216064:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762543.934194:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762544.958219:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762545.982213:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762547.006370:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762548.030226:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762548.414165:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762548.414180:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762548.414186:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762548.414197:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2474 00000100:00080000:1.0:1789762548.414210:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762548.414218:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762548.414266:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504216192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762548.414294:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762548.414298:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762548.414299:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762548.414303:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2474 00000100:00080000:1.0:1789762548.414306:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762548.414309:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762548.414320:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762548.414326:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504216192:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762548.414333:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504216320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762548.414353:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762548.414355:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762548.414356:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762548.414359:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2474 00000100:00080000:1.0:1789762548.414361:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762548.414363:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762548.414366:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762548.414374:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504216448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762548.414381:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762548.414384:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762548.414385:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762548.414388:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2474 00000100:00080000:1.0:1789762548.414390:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762548.414393:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762548.414397:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762548.414405:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504216576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762548.414412:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762548.414416:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1789762548.414417:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762548.414419:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762548.414425:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2474 00000100:00080000:1.0:1789762548.414428:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762548.414431:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762548.414434:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762548.414442:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504216704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762548.414465:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762548.414468:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762548.414470:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762548.414474:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2474 00000100:00080000:1.0:1789762548.414477:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762548.414480:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762548/real 1789762548] req@ffff9018c66a5c00 x1876699504216192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762548.414484:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762548.414489:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762548.414498:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504216832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762548.414510:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504216192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762548.414521:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762548.414526:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762548.414530:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504216192:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762548.414554:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504216320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762548.414573:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762548.414582:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762548/real 1789762548] req@ffff9018c66a6d80 x1876699504216320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762548.414590:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504216320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762548.414596:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762548.414598:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762548.414601:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504216320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762548.414604:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504216448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762548.414614:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762548.414620:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762548/real 1789762548] req@ffff9018c66a5f80 x1876699504216448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762548.414627:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504216448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762548.414632:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762548.414634:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762548.414636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504216448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762548.414639:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504216576:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762548.414651:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762548.414657:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762548/real 1789762548] req@ffff9018c66a4000 x1876699504216576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762548.414664:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504216576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762548.414669:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762548.414671:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762548.414673:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504216576:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762548.414676:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504216704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762548.414686:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762548.414692:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762548/real 1789762548] req@ffff9018c66a7b80 x1876699504216704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762548.414699:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504216704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762548.414704:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762548.414706:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762548.414707:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504216704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762548.414711:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504216832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762548.414721:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762548.414726:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762548/real 1789762548] req@ffff9018c66a7800 x1876699504216832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762548.414738:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504216832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762548.414743:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762548.414745:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762548.414746:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504216832:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762549.054199:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762550.078272:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762551.102250:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762552.126238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762553.150223:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762553.534224:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762553.534238:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762553.534244:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762553.534256:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2480 00000100:00080000:1.0:1789762553.534268:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762553.534275:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762553.534322:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504216960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762553.534350:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762553.534353:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762553.534354:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762553.534357:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2480 00000100:00080000:1.0:1789762553.534359:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762553.534362:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762553.534371:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762553.534381:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504217088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762553.534388:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762553.534390:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762553.534392:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762553.534395:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2480 00000100:00080000:1.0:1789762553.534396:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762553.534401:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762553.534404:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504216960:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789762553.534405:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762553.534433:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504217216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762553.534445:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762553.534448:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762553.534449:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762553.534453:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2480 00000100:00080000:1.0:1789762553.534455:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762553.534458:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762553.534462:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762553.534473:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504217344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762553.534479:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762553.534482:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762553.534483:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762553.534486:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2480 00000100:00080000:1.0:1789762553.534488:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762553.534491:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762553.534494:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762553.534498:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762553.534502:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504217472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762553.534510:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762553.534513:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762553.534514:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762553.534518:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2480 00000100:00080000:1.0:1789762553.534521:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762553.534524:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762553.534527:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762553.534533:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504217600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762553.534549:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762553/real 1789762553] req@ffff9018c66a5180 x1876699504216960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762553.534565:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504216960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762553.534576:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762553.534579:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762553.534582:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504216960:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762553.534589:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504217088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762553.534603:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762553.534611:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762553/real 1789762553] req@ffff9018c66a6300 x1876699504217088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762553.534619:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504217088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762553.534625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762553.534628:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762553.534630:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504217088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762553.534655:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504217216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762553.534666:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762553.534672:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762553/real 1789762553] req@ffff9018c66a4700 x1876699504217216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762553.534679:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504217216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762553.534684:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762553.534686:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762553.534688:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504217216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762553.534692:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504217344:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762553.534702:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762553.534715:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762553/real 1789762553] req@ffff9018c66a6680 x1876699504217344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762553.534723:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504217344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762553.534728:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762553.534730:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762553.534733:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504217344:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762553.534737:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504217472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762553.534749:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762553.534755:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762553/real 1789762553] req@ffff9018c66a5500 x1876699504217472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762553.534762:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504217472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762553.534767:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762553.534769:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762553.534771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504217472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762553.534774:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504217600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762553.534784:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762553.534789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762553/real 1789762553] req@ffff9018c66a4e00 x1876699504217600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762553.534796:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504217600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762553.534806:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762553.534808:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762553.534810:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504217600:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762554.174197:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762555.198182:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762556.222276:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762557.246177:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762558.270188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762558.654110:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762558.654121:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762558.654126:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762558.654136:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2485 00000100:00080000:1.0:1789762558.654147:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762558.654154:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762558.654196:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504217728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762558.654220:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762558.654223:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762558.654225:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762558.654228:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2485 00000100:00080000:1.0:1789762558.654230:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762558.654233:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762558.654242:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762558.654252:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504217856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762558.654260:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762558.654262:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762558.654264:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762558.654267:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2485 00000100:00080000:1.0:1789762558.654269:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762558.654275:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762558.654276:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504217728:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789762558.654279:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762558.654294:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504217984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762558.654300:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762558.654303:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762558.654304:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762558.654307:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2485 00000100:00080000:1.0:1789762558.654309:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762558.654311:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762558.654315:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762558.654323:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504218112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762558.654333:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762558.654337:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762558.654339:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762558.654343:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2485 00000100:00080000:1.0:1789762558.654348:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762558.654354:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762558.654360:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762558.654386:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504218240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762558.654394:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762558.654396:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762558.654398:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762558.654401:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2485 00000100:00080000:1.0:1789762558.654404:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762558.654406:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762558.654410:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762558.654418:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504218368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762558.654449:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504217856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762558.654483:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504217984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762558.654504:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504218112:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762558.654527:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504218240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762558.654545:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504218368:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762559.294168:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762560.318194:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762561.342161:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762561.726364:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789762561.726427:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762561.726450:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762561.726499:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789762561.726534:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762558/real 1789762561] req@ffff9018c66a4380 x1876699504217856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1789762561.726542:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762561.726571:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789762561.726578:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504217856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762561.726592:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:2.0:1789762561.726594:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00080000:3.0:1789762561.726606:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762561.726614:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504217856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762561.726627:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789762561.726642:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762558/real 1789762561] req@ffff9018c66a6a00 x1876699504217984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762561.726671:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504217984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762561.726682:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762561.726687:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762561.726691:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504217984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762561.726702:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762558/real 1789762561] req@ffff9018c66a7480 x1876699504218112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762561.726719:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504218112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762561.726729:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762561.726733:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762561.726738:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504218112:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762561.726750:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762558/real 1789762561] req@ffff9018c66a4a80 x1876699504218240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762561.726764:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504218240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762561.726774:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762561.726779:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762561.726783:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504218240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762561.726792:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762558/real 1789762561] req@ffff9018c66a7100 x1876699504218368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762561.726809:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504218368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762561.726817:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762561.726821:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762561.726825:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504218368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762561.726834:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762558/real 1789762561] req@ffff9018c66a5880 x1876699504217728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762561.726848:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504217728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762561.726857:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762561.726861:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762561.726865:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504217728:192.168.203.158@tcp:250:kworker.0 00010000:00010000:0.0:1789762562.366182:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762563.390248:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762563.774228:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762563.774236:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762563.774241:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762563.774251:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2490 00000100:00080000:0.0:1789762563.774272:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762563.774278:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762563.774313:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504218496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762563.774348:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762563.774352:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762563.774354:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762563.774357:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2490 00000100:00080000:0.0:1789762563.774361:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762563.774364:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762563.774383:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762563.774395:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504218624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762563.774409:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762563.774412:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762563.774413:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762563.774416:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2490 00000100:00080000:0.0:1789762563.774419:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762563.774426:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762563.774430:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762563.774431:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504218496:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762563.774444:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504218752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762563.774453:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762563.774457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762563.774458:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762563.774462:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2490 00000100:00080000:0.0:1789762563.774464:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762563.774468:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762563.774474:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762563.774489:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504218880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762563.774503:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762563.774507:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762563.774509:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762563.774514:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2490 00000100:00080000:0.0:1789762563.774520:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762563.774525:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762563.774530:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762563.774544:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504219008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762563.774544:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762563.774558:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762563.774560:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762563.774562:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762563.774574:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2490 00000100:00080000:0.0:1789762563.774581:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762563.774585:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762563.774589:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762563.774599:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504219136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762563.774636:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762563/real 1789762563] req@ffff9018c8b9b100 x1876699504218496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762563.774667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504218496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762563.774711:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762563.774717:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762563.774724:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504218496:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762563.774765:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504218624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762563.774803:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762563.774824:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762563/real 1789762563] req@ffff9018c8b9b480 x1876699504218624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762563.774843:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504218624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762563.774855:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762563.774860:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762563.774864:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504218624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762563.774874:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504218752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762563.774906:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762563.774924:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762563/real 1789762563] req@ffff9018c8b98000 x1876699504218752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762563.774941:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504218752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762563.774953:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762563.774958:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762563.774963:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504218752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762563.774974:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504218880:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762563.775006:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762563.775024:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762563/real 1789762563] req@ffff9018c8b9aa00 x1876699504218880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762563.775090:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504218880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762563.775104:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762563.775109:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762563.775115:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504218880:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762563.775142:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504219008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762563.775175:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762563.775353:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762563/real 1789762563] req@ffff9018c8b98e00 x1876699504219008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762563.775373:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504219008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762563.775386:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762563.775391:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762563.775395:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504219008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762563.775406:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504219136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762563.775439:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762563.775456:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762563/real 1789762563] req@ffff9018c8b9a680 x1876699504219136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762563.775473:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504219136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762563.775484:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762563.775488:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762563.775492:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504219136:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762564.414213:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762565.438225:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762566.462188:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762567.486253:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762568.510231:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762568.894112:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762568.894121:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762568.894125:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762568.894134:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2495 00000100:00080000:0.0:1789762568.894144:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762568.894150:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762568.894183:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504219264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762568.894215:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762568.894220:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762568.894222:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762568.894227:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2495 00000100:00080000:0.0:1789762568.894231:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762568.894236:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762568.894246:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762568.894263:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504219392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762568.894272:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762568.894275:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762568.894277:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762568.894281:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2495 00000100:00080000:0.0:1789762568.894288:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762568.894291:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762568.894309:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762568.894323:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504219520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762568.894340:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762568.894340:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504219264:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762568.894358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762568.894360:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762568.894366:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2495 00000100:00080000:0.0:1789762568.894371:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762568.894377:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762568.894384:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762568.894397:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504219648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762568.894408:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762568.894411:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762568.894412:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762568.894417:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2495 00000100:00080000:0.0:1789762568.894423:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762568.894426:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762568.894429:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762568.894439:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762568.894445:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504219776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762568.894454:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762568.894457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762568.894459:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762568.894463:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2495 00000100:00080000:0.0:1789762568.894466:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762568.894468:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762568.894472:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762568.894481:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504219904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762568.894495:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762568/real 1789762568] req@ffff9018c8b9a300 x1876699504219264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762568.894514:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504219264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762568.894525:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762568.894529:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762568.894532:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504219264:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762568.894541:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504219392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762568.894560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762568.894569:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762568/real 1789762568] req@ffff9018c8b9ad80 x1876699504219392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762568.894577:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504219392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762568.894584:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762568.894586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762568.894589:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504219392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762568.894629:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504219520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762568.894646:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762568.894655:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762568/real 1789762568] req@ffff9018c8b99500 x1876699504219520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762568.894667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504219520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762568.894673:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762568.894676:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762568.894679:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504219520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762568.894685:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504219648:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762568.894711:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762568.894731:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762568/real 1789762568] req@ffff9018c8b99c00 x1876699504219648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762568.894741:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504219648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762568.894747:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762568.894749:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762568.894751:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504219648:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762568.894756:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504219776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762568.894767:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762568.894774:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762568/real 1789762568] req@ffff9018c8b98380 x1876699504219776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762568.894781:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504219776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762568.894786:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762568.894788:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762568.894790:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504219776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762568.894793:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504219904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762568.894803:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762568.894809:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762568/real 1789762568] req@ffff9018c8b9a680 x1876699504219904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762568.894827:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504219904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762568.894836:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762568.894842:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762568.894844:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504219904:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762569.534341:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762570.558204:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762571.582252:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762572.606360:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762573.630296:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762574.014181:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762574.014195:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762574.014201:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762574.014216:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2500 00000100:00080000:2.0:1789762574.014232:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762574.014242:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762574.014327:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504220032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762574.014371:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762574.014377:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762574.014380:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762574.014385:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2500 00000100:00080000:2.0:1789762574.014390:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762574.014394:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762574.014408:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762574.014425:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504220160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762574.014449:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762574.014455:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762574.014461:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762574.014463:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504220032:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762574.014469:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2500 00000100:00080000:2.0:1789762574.014479:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762574.014485:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762574.014492:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762574.014509:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504220288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762574.014521:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762574.014526:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762574.014529:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762574.014533:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2500 00000100:00080000:2.0:1789762574.014564:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762574.014569:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762574.014576:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789762574.014585:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762574.014598:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504220416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762574.014614:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762574.014619:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762574.014622:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762574.014626:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2500 00000100:00080000:2.0:1789762574.014631:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762574.014637:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762574.014644:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762574.014657:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762574/real 1789762574] req@ffff9018d0f25500 x1876699504220032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762574.014664:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504220544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762574.014715:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762574.014720:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762574.014723:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762574.014732:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2500 00000100:00100000:3.0:1789762574.014733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504220032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762574.014741:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762574.014753:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762574.014765:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789762574.014772:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789762574.014781:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504220672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762574.014785:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762574.014799:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504220032:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762574.014847:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504220160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762574.014883:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762574.014899:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762574/real 1789762574] req@ffff9018d0f27480 x1876699504220160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762574.014940:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504220160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762574.014958:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762574.014964:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762574.014972:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504220160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762574.014985:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504220288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762574.015027:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762574.015109:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762574/real 1789762574] req@ffff9018d0f26680 x1876699504220288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762574.015126:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504220288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762574.015136:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762574.015141:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762574.015145:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504220288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762574.015152:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504220416:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762574.015181:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762574.015194:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762574/real 1789762574] req@ffff9018d0f24700 x1876699504220416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762574.015218:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504220416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762574.015228:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762574.015232:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762574.015235:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504220416:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762574.015242:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504220544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762574.015296:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762574.015312:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762574/real 1789762574] req@ffff9018d0f24e00 x1876699504220544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762574.015328:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504220544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762574.015339:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762574.015343:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762574.015347:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504220544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762574.015354:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504220672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762574.015375:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762574.015387:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762574/real 1789762574] req@ffff9018d0f24000 x1876699504220672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762574.015401:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504220672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762574.015418:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762574.015422:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762574.015426:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504220672:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762574.654169:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762575.678238:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762576.702279:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762577.726202:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762578.750163:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762579.134149:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762579.134157:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762579.134160:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762579.134168:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2505 00000100:00080000:0.0:1789762579.134177:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762579.134183:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762579.134231:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504220800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762579.134256:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762579.134260:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762579.134261:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762579.134264:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2505 00000100:00080000:0.0:1789762579.134267:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762579.134270:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762579.134279:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762579.134289:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504220928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762579.134297:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762579.134300:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762579.134301:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762579.134305:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2505 00000100:00080000:0.0:1789762579.134307:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762579.134310:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762579.134314:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762579.134315:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504220800:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762579.134323:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504221056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762579.134330:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762579.134333:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762579.134334:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762579.134337:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2505 00000100:00080000:0.0:1789762579.134339:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762579.134341:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762579.134346:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762579.134354:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504221184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762579.134362:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762579.134364:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762579.134366:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762579.134368:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2505 00000100:00080000:0.0:1789762579.134370:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789762579.134375:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762579.134377:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762579.134382:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762579.134390:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504221312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762579.134398:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762579.134399:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762579.134400:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762579.134403:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2505 00000100:00080000:0.0:1789762579.134405:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762579.134408:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762579.134411:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762579.134416:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762579/real 1789762579] req@ffff9018c8b98e00 x1876699504220800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762579.134421:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504221440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762579.134457:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504220800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762579.134467:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762579.134470:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762579.134474:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504220800:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762579.134480:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504220928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762579.134494:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762579.134502:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762579/real 1789762579] req@ffff9018c8b9aa00 x1876699504220928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762579.134510:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504220928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762579.134530:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762579.134532:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762579.134535:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504220928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762579.134552:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504221056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762579.134565:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762579.134571:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762579/real 1789762579] req@ffff9018c8b98000 x1876699504221056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762579.134579:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504221056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762579.134584:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762579.134586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762579.134588:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504221056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762579.134592:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504221184:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762579.134602:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762579.134607:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762579/real 1789762579] req@ffff9018c8b9b480 x1876699504221184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762579.134614:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504221184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762579.134619:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762579.134621:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762579.134623:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504221184:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762579.134627:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504221312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762579.134636:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762579.134642:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762579/real 1789762579] req@ffff9018c8b9b100 x1876699504221312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762579.134649:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504221312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762579.134654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762579.134656:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762579.134657:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504221312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762579.134661:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504221440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762579.134670:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762579.134675:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762579/real 1789762579] req@ffff9018c8b9b800 x1876699504221440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762579.134682:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504221440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762579.134687:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762579.134688:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762579.134690:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504221440:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762579.774187:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762580.798166:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762581.822167:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762582.846187:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762583.870249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762584.254150:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762584.254161:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762584.254165:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762584.254176:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2510 00000100:00080000:2.0:1789762584.254201:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762584.254209:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762584.254253:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504221568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762584.254280:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762584.254286:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762584.254288:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762584.254293:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2510 00000100:00080000:2.0:1789762584.254296:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762584.254302:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762584.254315:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762584.254330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504221696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762584.254338:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504221568:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762584.254340:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762584.254344:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762584.254346:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762584.254350:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2510 00000100:00080000:2.0:1789762584.254353:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762584.254357:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762584.254361:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762584.254371:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504221824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762584.254379:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762584.254383:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762584.254384:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762584.254388:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2510 00000100:00080000:2.0:1789762584.254400:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762584.254404:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762584.254410:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789762584.254411:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762584.254421:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504221952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762584.254432:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762584.254435:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762584.254437:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762584.254440:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2510 00000100:00080000:2.0:1789762584.254444:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762584.254447:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762584.254452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762584.254461:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504222080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762584.254470:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762584.254473:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762584.254475:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762584.254476:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762584/real 1789762584] req@ffff9018d0f24a80 x1876699504221568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762584.254479:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2510 00000100:00080000:2.0:1789762584.254482:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762584.254486:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762584.254490:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762584.254497:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504221568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762584.254498:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504222208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762584.254511:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762584.254515:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762584.254519:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504221568:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762584.254546:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504221696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762584.254581:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762584.254597:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762584/real 1789762584] req@ffff9018d0f25180 x1876699504221696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762584.254612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504221696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762584.254621:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762584.254625:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762584.254628:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504221696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762584.254633:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504221824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762584.254650:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762584.254659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762584/real 1789762584] req@ffff9018d0f27800 x1876699504221824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762584.254668:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504221824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762584.254673:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762584.254675:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762584.254678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504221824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762584.254681:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504221952:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762584.254701:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762584.254708:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762584/real 1789762584] req@ffff9018d0f24380 x1876699504221952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762584.254716:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504221952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762584.254722:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762584.254724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762584.254726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504221952:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762584.254734:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504222080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762584.254745:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762584.254752:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762584/real 1789762584] req@ffff9018d0f26a00 x1876699504222080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762584.254760:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504222080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762584.254770:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762584.254772:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762584.254774:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504222080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762584.254778:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504222208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762584.254787:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762584.254793:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762584/real 1789762584] req@ffff9018d0f25c00 x1876699504222208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762584.254800:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504222208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762584.254811:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762584.254813:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762584.254816:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504222208:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762584.894238:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762585.918238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762586.942258:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762587.966210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762588.990277:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762589.374161:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762589.374172:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762589.374177:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762589.374206:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2515 00000100:00080000:2.0:1789762589.374218:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762589.374226:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762589.374269:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504222336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762589.374295:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762589.374298:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762589.374300:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762589.374303:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2515 00000100:00080000:2.0:1789762589.374305:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762589.374307:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762589.374316:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762589.374325:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504222464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762589.374333:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762589.374336:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762589.374337:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762589.374340:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2515 00000100:00080000:2.0:1789762589.374342:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762589.374344:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762589.374348:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762589.374355:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504222336:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762589.374357:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504222592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762589.374374:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762589.374376:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762589.374377:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762589.374380:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2515 00000100:00080000:2.0:1789762589.374382:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762589.374385:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762589.374388:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762589.374397:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504222720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762589.374404:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762589.374408:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762589.374409:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762589.374411:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2515 00000100:00080000:2.0:1789762589.374421:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762589.374424:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762589.374426:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789762589.374433:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762589.374438:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504222848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762589.374446:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762589.374449:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762589.374451:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762589.374454:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2515 00000100:00080000:2.0:1789762589.374458:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762589.374461:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762589.374464:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762589.374472:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504222976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762589.374482:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762589/real 1789762589] req@ffff9018d0f26300 x1876699504222336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762589.374501:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504222336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762589.374515:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762589.374519:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762589.374522:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504222336:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762589.374528:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504222464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762589.374543:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762589.374552:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762589/real 1789762589] req@ffff9018d0f27100 x1876699504222464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762589.374560:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504222464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762589.374566:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762589.374568:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762589.374570:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504222464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762589.374592:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504222592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762589.374603:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762589.374609:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762589/real 1789762589] req@ffff9018d0f25f80 x1876699504222592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762589.374617:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504222592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762589.374622:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762589.374625:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762589.374626:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504222592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762589.374631:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504222720:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762589.374641:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762589.374647:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762589/real 1789762589] req@ffff9018d0f26d80 x1876699504222720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762589.374664:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504222720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762589.374670:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762589.374672:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762589.374674:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504222720:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762589.374678:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504222848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762589.374690:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762589.374696:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762589/real 1789762589] req@ffff9018d0f25c00 x1876699504222848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762589.374703:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504222848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762589.374708:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762589.374710:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762589.374712:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504222848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762589.374715:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504222976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762589.374733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762589.374739:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762589/real 1789762589] req@ffff9018d0f26a00 x1876699504222976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762589.374748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504222976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762589.374753:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762589.374755:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762589.374757:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504222976:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762590.014216:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762591.038246:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762592.062192:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762593.086302:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762594.110193:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762594.494140:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762594.494149:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762594.494152:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762594.494161:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2521 00000100:00080000:2.0:1789762594.494170:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762594.494176:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762594.494206:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504223104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762594.494228:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762594.494231:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762594.494232:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762594.494235:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2521 00000100:00080000:2.0:1789762594.494237:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762594.494240:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762594.494247:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762594.494256:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504223232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762594.494263:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762594.494267:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762594.494268:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762594.494271:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2521 00000100:00080000:2.0:1789762594.494274:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762594.494276:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762594.494280:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762594.494281:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504223104:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762594.494288:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504223360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762594.494296:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762594.494298:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762594.494299:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762594.494301:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2521 00000100:00080000:2.0:1789762594.494304:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762594.494306:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762594.494310:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762594.494317:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504223488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762594.494324:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762594.494327:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762594.494328:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762594.494331:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2521 00000100:00080000:2.0:1789762594.494333:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762594.494335:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762594.494339:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789762594.494342:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762594.494349:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504223616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762594.494357:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762594.494359:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762594.494360:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762594.494363:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2521 00000100:00080000:2.0:1789762594.494365:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762594.494367:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762594.494370:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762594.494378:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504223744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762594.494383:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762594/real 1789762594] req@ffff9018d0f24380 x1876699504223104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762594.494400:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504223104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762594.494409:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762594.494412:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762594.494416:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504223104:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762594.494421:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504223232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762594.494435:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762594.494443:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762594/real 1789762594] req@ffff9018d0f27800 x1876699504223232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762594.494455:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504223232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762594.494461:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762594.494462:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762594.494464:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504223232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762594.494483:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504223360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762594.494494:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762594.494500:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762594/real 1789762594] req@ffff9018d0f25180 x1876699504223360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762594.494507:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504223360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762594.494512:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762594.494514:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762594.494516:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504223360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762594.494519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504223488:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762594.494529:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762594.494544:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762594/real 1789762594] req@ffff9018d0f24a80 x1876699504223488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762594.494552:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504223488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762594.494562:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762594.494564:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762594.494566:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504223488:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762594.494570:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504223616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762594.494580:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762594.494587:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762594/real 1789762594] req@ffff9018d0f24000 x1876699504223616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762594.494594:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504223616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762594.494599:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762594.494601:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762594.494603:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504223616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762594.494606:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504223744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762594.494615:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762594.494621:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762594/real 1789762594] req@ffff9018d0f24e00 x1876699504223744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762594.494628:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504223744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762594.494632:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762594.494634:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762594.494636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504223744:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762595.134162:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762596.158301:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762597.182215:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762598.206265:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762599.230254:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762599.614176:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762599.614187:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762599.614192:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762599.614202:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2526 00000100:00080000:2.0:1789762599.614215:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762599.614222:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762599.614269:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504223872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762599.614298:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762599.614302:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762599.614304:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762599.614308:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2526 00000100:00080000:2.0:1789762599.614313:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762599.614319:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762599.614331:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762599.614346:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504224000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762599.614356:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762599.614359:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762599.614361:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762599.614361:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504223872:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762599.614365:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2526 00000100:00080000:2.0:1789762599.614369:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762599.614372:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762599.614377:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762599.614386:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504224128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762599.614395:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762599.614398:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762599.614400:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762599.614403:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2526 00000100:00080000:2.0:1789762599.614406:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762599.614417:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762599.614421:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762599.614431:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504224256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762599.614443:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762599.614446:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762599.614449:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762599.614451:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762599.614454:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2526 00000100:00080000:2.0:1789762599.614457:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762599.614460:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762599.614464:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762599.614475:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504224384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762599.614483:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762599.614486:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762599.614488:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762599.614491:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2526 00000100:00100000:3.0:1789762599.614493:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762599/real 1789762599] req@ffff9018d0f24700 x1876699504223872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762599.614495:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762599.614499:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762599.614504:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762599.614512:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504224512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762599.614515:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504223872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762599.614527:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762599.614531:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762599.614535:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504223872:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762599.614564:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504224000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762599.614589:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762599.614600:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762599/real 1789762599] req@ffff9018d0f26680 x1876699504224000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762599.614609:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504224000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762599.614615:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762599.614618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762599.614620:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504224000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762599.614626:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504224128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762599.614639:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762599.614645:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762599/real 1789762599] req@ffff9018d0f27480 x1876699504224128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762599.614653:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504224128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762599.614658:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762599.614660:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762599.614662:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504224128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762599.614665:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504224256:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762599.614676:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762599.614682:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762599/real 1789762599] req@ffff9018d0f25500 x1876699504224256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762599.614689:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504224256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762599.614696:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762599.614700:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762599.614704:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504224256:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762599.614710:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504224384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762599.614730:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762599.614743:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762599/real 1789762599] req@ffff9018d0f24e00 x1876699504224384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762599.614756:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504224384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762599.614762:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762599.614765:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762599.614767:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504224384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762599.614771:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504224512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762599.614786:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762599.614793:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762599/real 1789762599] req@ffff9018d0f24000 x1876699504224512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762599.614801:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504224512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762599.614806:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762599.614808:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762599.614810:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504224512:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762600.254186:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762601.278152:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762602.302156:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762603.326200:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762604.350205:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762604.734140:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762604.734156:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762604.734160:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762604.734171:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2531 00000100:00080000:2.0:1789762604.734184:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762604.734213:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762604.734263:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504224640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762604.734295:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762604.734299:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762604.734301:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762604.734307:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2531 00000100:00080000:2.0:1789762604.734311:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762604.734317:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762604.734331:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762604.734336:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504224640:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762604.734355:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504224768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762604.734372:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762604.734376:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762604.734378:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762604.734382:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2531 00000100:00080000:2.0:1789762604.734387:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762604.734390:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762604.734394:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762604.734407:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504224896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762604.734419:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762604.734426:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762604.734427:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762604.734433:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762604.734439:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2531 00000100:00080000:2.0:1789762604.734445:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762604.734451:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762604.734457:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762604.734474:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504225024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762604.734480:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762604/real 1789762604] req@ffff9018d0f24a80 x1876699504224640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762604.734497:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762604.734501:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762604.734503:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762604.734510:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504224640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762604.734510:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2531 00000100:00080000:2.0:1789762604.734515:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762604.734523:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789762604.734528:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789762604.734529:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762604.734536:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762604.734541:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504224640:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762604.734545:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504225152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762604.734554:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762604.734557:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762604.734558:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762604.734561:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2531 00000100:00080000:2.0:1789762604.734562:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762604.734565:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762604.734570:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762604.734571:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504224768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762604.734584:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504225280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762604.734600:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762604.734618:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762604/real 1789762604] req@ffff9018d0f25180 x1876699504224768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762604.734628:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504224768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762604.734642:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762604.734645:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762604.734650:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504224768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762604.734656:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504224896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762604.734676:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762604.734684:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762604/real 1789762604] req@ffff9018d0f27800 x1876699504224896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762604.734699:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504224896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762604.734729:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762604.734734:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762604.734739:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504224896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762604.734749:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504225024:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762604.734774:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762604.734787:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762604/real 1789762604] req@ffff9018d0f24380 x1876699504225024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762604.734806:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504225024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762604.734815:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762604.734819:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762604.734822:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504225024:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762604.734827:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504225152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762604.734850:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762604.734862:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762604/real 1789762604] req@ffff9018d0f26a00 x1876699504225152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762604.734874:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504225152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762604.734882:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762604.734885:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762604.734888:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504225152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762604.734907:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504225280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762604.734928:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762604.734941:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762604/real 1789762604] req@ffff9018d0f25c00 x1876699504225280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762604.734953:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504225280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762604.734967:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762604.734970:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762604.734973:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504225280:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762605.374150:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762606.398229:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762607.422195:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762608.446197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762609.470174:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762609.854153:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762609.854167:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762609.854173:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762609.854187:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2536 00000100:00080000:2.0:1789762609.854200:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762609.854211:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762609.854260:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504225408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762609.854299:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762609.854308:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762609.854310:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762609.854316:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2536 00000100:00080000:2.0:1789762609.854320:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762609.854325:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762609.854337:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762609.854353:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504225536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762609.854360:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504225408:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762609.854372:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762609.854377:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762609.854379:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762609.854385:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2536 00000100:00080000:2.0:1789762609.854390:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762609.854395:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762609.854401:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762609.854416:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504225664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762609.854427:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762609.854431:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762609.854433:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789762609.854438:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762609.854442:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2536 00000100:00080000:2.0:1789762609.854448:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762609.854453:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762609.854460:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762609.854474:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504225792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762609.854486:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762609.854491:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762609.854493:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762609.854499:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762609/real 1789762609] req@ffff9018d0f26d80 x1876699504225408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762609.854501:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2536 00000100:00080000:2.0:1789762609.854510:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762609.854519:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762609.854526:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762609.854532:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504225408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762609.854542:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504225920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762609.854547:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762609.854555:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762609.854560:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504225408:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762609.854576:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762609.854580:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762609.854583:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762609.854588:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2536 00000100:00080000:2.0:1789762609.854593:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762609.854597:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762609.854603:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762609.854607:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504225536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762609.854627:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504226048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762609.854644:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762609.854657:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762609/real 1789762609] req@ffff9018d0f25f80 x1876699504225536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762609.854667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504225536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762609.854674:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762609.854676:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762609.854678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504225536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762609.854683:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504225664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762609.854696:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762609.854702:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762609/real 1789762609] req@ffff9018d0f27100 x1876699504225664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762609.854710:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504225664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762609.854714:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762609.854716:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762609.854718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504225664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762609.854722:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504225792:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762609.854733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762609.854739:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762609/real 1789762609] req@ffff9018d0f26300 x1876699504225792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762609.854747:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504225792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762609.854752:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762609.854754:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762609.854756:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504225792:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762609.854759:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504225920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762609.854768:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762609.854774:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762609/real 1789762609] req@ffff9018d0f25c00 x1876699504225920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762609.854782:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504225920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762609.854786:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762609.854788:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762609.854790:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504225920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762609.854802:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504226048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762609.854822:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762609.854833:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762609/real 1789762609] req@ffff9018d0f26a00 x1876699504226048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762609.854840:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504226048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762609.854848:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762609.854851:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762609.854854:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504226048:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762610.494255:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762611.518190:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762612.542228:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762613.566189:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762614.590151:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762614.974125:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762614.974136:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762614.974140:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762614.974152:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2541 00000100:00080000:2.0:1789762614.974164:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762614.974172:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762614.974214:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504226176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762614.974241:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762614.974244:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762614.974246:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762614.974249:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2541 00000100:00080000:2.0:1789762614.974252:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762614.974255:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762614.974266:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762614.974278:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504226304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762614.974283:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504226176:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762614.974289:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762614.974293:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762614.974294:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762614.974297:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2541 00000100:00080000:2.0:1789762614.974299:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762614.974301:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762614.974304:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762614.974313:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504226432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762614.974319:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762614.974322:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762614.974323:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762614.974325:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2541 00000100:00080000:2.0:1789762614.974327:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762614.974330:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762614.974335:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762614.974342:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504226560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762614.974349:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762614.974352:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762614.974353:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762614.974356:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2541 00000100:00080000:2.0:1789762614.974359:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762614.974362:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762614.974365:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762614.974369:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762614.974376:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504226688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762614.974390:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762614.974393:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762614.974394:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762614.974397:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2541 00000100:00080000:2.0:1789762614.974400:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762614.974402:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762614.974405:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762614.974412:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504226816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762614.974427:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762614/real 1789762614] req@ffff9018d0f24380 x1876699504226176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762614.974451:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504226176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762614.974471:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762614.974474:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762614.974478:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504226176:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762614.974502:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504226304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762614.974522:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762614.974532:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762614/real 1789762614] req@ffff9018d0f27800 x1876699504226304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762614.974540:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504226304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762614.974554:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762614.974557:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762614.974559:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504226304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762614.974563:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504226432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762614.974577:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762614.974584:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762614/real 1789762614] req@ffff9018d0f25180 x1876699504226432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762614.974591:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504226432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762614.974596:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762614.974598:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762614.974600:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504226432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762614.974604:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504226560:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762614.974615:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762614.974621:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762614/real 1789762614] req@ffff9018d0f24a80 x1876699504226560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762614.974629:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504226560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762614.974634:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762614.974636:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762614.974638:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504226560:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762614.974641:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504226688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762614.974651:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762614.974656:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762614/real 1789762614] req@ffff9018d0f24000 x1876699504226688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762614.974663:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504226688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762614.974667:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762614.974670:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762614.974672:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504226688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762614.974675:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504226816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762614.974685:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762614.974690:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762614/real 1789762614] req@ffff9018d0f24e00 x1876699504226816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762614.974697:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504226816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762614.974702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762614.974704:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762614.974706:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504226816:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762615.614180:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762616.638189:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762617.662176:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762618.686313:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762619.710190:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762620.094156:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762620.094167:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762620.094172:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762620.094183:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2546 00000100:00080000:2.0:1789762620.094195:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762620.094202:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762620.094246:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504226944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762620.094277:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762620.094281:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762620.094282:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762620.094285:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2546 00000100:00080000:2.0:1789762620.094287:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762620.094291:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762620.094302:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762620.094311:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504227072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762620.094319:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762620.094323:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762620.094327:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762620.094330:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504226944:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762620.094334:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2546 00000100:00080000:2.0:1789762620.094344:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762620.094350:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762620.094353:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762620.094360:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504227200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762620.094367:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762620.094370:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762620.094372:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762620.094376:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2546 00000100:00080000:2.0:1789762620.094379:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762620.094390:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762620.094396:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762620.094404:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504227328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762620.094412:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762620.094415:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762620.094417:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789762620.094426:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762620.094431:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2546 00000100:00080000:2.0:1789762620.094437:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762620.094441:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762620.094445:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762620.094461:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504227456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762620.094468:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762620.094474:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762620/real 1789762620] req@ffff9018d0f25500 x1876699504226944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762620.094474:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762620.094479:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762620.094494:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2546 00000100:00080000:2.0:1789762620.094503:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762620.094509:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762620.094512:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504226944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789762620.094518:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762620.094524:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762620.094529:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762620.094539:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504226944:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762620.094541:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504227584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762620.094581:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504227072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762620.094610:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762620.094620:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762620/real 1789762620] req@ffff9018d0f27480 x1876699504227072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762620.094637:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504227072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762620.094649:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762620.094651:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762620.094654:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504227072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762620.094658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504227200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762620.094671:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762620.094679:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762620/real 1789762620] req@ffff9018d0f26680 x1876699504227200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762620.094686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504227200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762620.094692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762620.094695:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762620.094697:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504227200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762620.094700:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504227328:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762620.094711:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762620.094717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762620/real 1789762620] req@ffff9018d0f24700 x1876699504227328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762620.094725:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504227328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762620.094730:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762620.094732:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762620.094735:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504227328:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762620.094739:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504227456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762620.094749:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762620.094755:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762620/real 1789762620] req@ffff9018d0f24e00 x1876699504227456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762620.094762:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504227456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762620.094767:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762620.094769:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762620.094771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504227456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762620.094775:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504227584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762620.094785:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762620.094790:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762620/real 1789762620] req@ffff9018d0f24000 x1876699504227584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762620.094797:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504227584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762620.094802:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762620.094804:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762620.094806:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504227584:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762620.734180:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762621.758272:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762622.782194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762623.806158:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762624.830199:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762625.214104:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762625.214115:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762625.214120:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762625.214130:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2551 00000100:00080000:2.0:1789762625.214142:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762625.214150:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762625.214197:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504227712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762625.214223:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762625.214227:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762625.214229:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762625.214235:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2551 00000100:00080000:2.0:1789762625.214238:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762625.214243:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762625.214255:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762625.214267:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504227712:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762625.214269:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504227840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762625.214279:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762625.214282:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762625.214284:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762625.214288:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2551 00000100:00080000:2.0:1789762625.214291:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762625.214295:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762625.214300:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762625.214309:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504227968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762625.214317:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762625.214321:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762625.214322:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762625.214326:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2551 00000100:00080000:2.0:1789762625.214329:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762625.214332:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762625.214337:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762625.214345:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504228096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762625.214355:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762625.214358:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762625.214360:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762625.214363:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2551 00000100:00080000:2.0:1789762625.214366:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762625.214370:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762625.214376:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762625.214387:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504228224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762625.214395:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762625.214399:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504227840:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789762625.214399:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762625.214401:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762625.214405:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2551 00000100:00080000:2.0:1789762625.214408:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762625.214412:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762625.214417:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762625.214427:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504228352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762625.214433:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504227968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762625.214458:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504228096:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762625.214486:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504228224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762625.214510:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504228352:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762625.854187:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762626.878240:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762627.902196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762628.286322:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:3.0:1789762628.286376:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762628.286394:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762628.286433:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762628.286441:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762628.286447:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762628.286452:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789762628.286459:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789762628.286501:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762625/real 1789762628] req@ffff9018d0f25c00 x1876699504228352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762628.286533:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504228352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762628.286547:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762628.286553:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762628.286558:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504228352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762628.286568:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762625/real 1789762628] req@ffff9018d0f25180 x1876699504227840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762628.286582:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504227840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762628.286589:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762628.286593:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762628.286596:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504227840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762628.286602:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762625/real 1789762628] req@ffff9018d0f27800 x1876699504227968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762628.286619:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504227968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762628.286628:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762628.286631:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762628.286634:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504227968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762628.286641:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762625/real 1789762628] req@ffff9018d0f24380 x1876699504228096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762628.286652:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504228096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762628.286659:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762628.286662:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762628.286665:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504228096:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762628.286673:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762625/real 1789762628] req@ffff9018d0f26a00 x1876699504228224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762628.286684:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504228224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762628.286691:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762628.286694:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762628.286697:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504228224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762628.286703:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762625/real 1789762628] req@ffff9018d0f24a80 x1876699504227712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762628.286716:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504227712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762628.286724:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762628.286727:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762628.286729:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504227712:192.168.203.158@tcp:250:kworker.0 00010000:00010000:3.0:1789762628.926244:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762629.950248:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762630.334115:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762630.334129:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762630.334135:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762630.334149:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2556 00000100:00080000:2.0:1789762630.334163:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762630.334172:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762630.334219:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504228480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762630.334255:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762630.334260:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762630.334262:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762630.334267:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2556 00000100:00080000:2.0:1789762630.334270:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762630.334278:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762630.334291:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762630.334307:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504228608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762630.334326:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762630.334334:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789762630.334336:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504228480:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762630.334337:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762630.334350:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2556 00000100:00080000:2.0:1789762630.334356:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762630.334362:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762630.334369:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762630.334385:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504228736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762630.334397:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762630.334401:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762630.334403:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762630.334409:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2556 00000100:00080000:2.0:1789762630.334415:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762630.334424:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789762630.334426:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789762630.334436:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762630.334457:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504228864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762630.334471:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762630.334475:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762630.334478:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762630.334487:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2556 00000100:00100000:1.0:1789762630.334488:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762630/real 1789762630] req@ffff9018d0f26300 x1876699504228480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762630.334498:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762630.334512:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762630.334521:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762630.334527:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504228480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762630.334543:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504228992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762630.334544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762630.334563:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789762630.334577:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762630.334585:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504228480:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762630.334588:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762630.334601:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762630.334610:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2556 00000100:00080000:2.0:1789762630.334617:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762630.334622:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762630.334631:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762630.334635:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504228608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762630.334652:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504229120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762630.334682:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762630.334708:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762630/real 1789762630] req@ffff9018d0f27100 x1876699504228608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762630.334724:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504228608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762630.334734:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762630.334740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762630.334744:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504228608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762630.334751:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504228736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762630.334774:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762630.334785:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762630/real 1789762630] req@ffff9018d0f25f80 x1876699504228736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762630.334799:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504228736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762630.334807:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762630.334811:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762630.334815:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504228736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762630.334823:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504228864:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762630.334862:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762630.334875:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762630/real 1789762630] req@ffff9018d0f26d80 x1876699504228864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762630.334895:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504228864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762630.334905:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762630.334909:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762630.334912:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504228864:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762630.334921:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504228992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762630.334943:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762630.334956:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762630/real 1789762630] req@ffff9018d0f24a80 x1876699504228992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762630.334970:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504228992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762630.334980:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762630.334984:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762630.334987:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504228992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762630.335008:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504229120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762630.335031:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762630.335082:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762630/real 1789762630] req@ffff9018d0f26a00 x1876699504229120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762630.335096:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504229120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762630.335105:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762630.335110:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762630.335114:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504229120:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762630.974212:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762631.998251:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762633.022182:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762634.046263:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762635.070181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762635.454140:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762635.454151:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762635.454156:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762635.454166:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2562 00000100:00080000:0.0:1789762635.454178:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762635.454185:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762635.454231:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504229248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762635.454256:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762635.454259:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762635.454260:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762635.454263:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2562 00000100:00080000:0.0:1789762635.454266:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762635.454268:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762635.454278:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762635.454290:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504229376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762635.454297:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762635.454300:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762635.454301:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762635.454304:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2562 00000100:00080000:0.0:1789762635.454306:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762635.454308:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762635.454311:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762635.454319:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504229248:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762635.454322:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504229504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762635.454333:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762635.454335:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762635.454336:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762635.454339:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2562 00000100:00080000:0.0:1789762635.454340:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762635.454343:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762635.454346:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762635.454354:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504229632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762635.454361:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762635.454364:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762635.454366:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762635.454375:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2562 00000100:00080000:0.0:1789762635.454384:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762635.454387:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762635.454391:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762635.454398:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762635.454402:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504229760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762635.454409:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762635.454412:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762635.454414:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762635.454417:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2562 00000100:00080000:0.0:1789762635.454420:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762635.454423:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762635.454426:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762635.454434:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504229888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762635.454451:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762635/real 1789762635] req@ffff9018c8b99880 x1876699504229248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762635.454481:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504229248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762635.454495:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762635.454500:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762635.454503:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504229248:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762635.454510:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504229376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762635.454529:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762635.454539:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762635/real 1789762635] req@ffff9018c8b98a80 x1876699504229376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762635.454547:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504229376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762635.454552:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762635.454554:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762635.454557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504229376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762635.454578:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504229504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762635.454589:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762635.454595:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762635/real 1789762635] req@ffff9018c8b99f80 x1876699504229504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762635.454603:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504229504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762635.454612:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762635.454614:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762635.454616:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504229504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762635.454620:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504229632:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762635.454630:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762635.454635:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762635/real 1789762635] req@ffff9018c8b98700 x1876699504229632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762635.454642:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504229632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762635.454648:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762635.454649:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762635.454651:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504229632:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762635.454654:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504229760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762635.454664:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762635.454669:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762635/real 1789762635] req@ffff9018c8b9bb80 x1876699504229760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762635.454676:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504229760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762635.454681:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762635.454683:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762635.454685:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504229760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762635.454689:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504229888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762635.454698:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762635.454704:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762635/real 1789762635] req@ffff9018c8b9b800 x1876699504229888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762635.454711:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504229888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762635.454715:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762635.454717:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762635.454719:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504229888:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762636.094257:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762637.118279:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762638.142208:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762639.166220:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762640.190241:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762640.574173:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762640.574181:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762640.574185:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762640.574194:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2567 00000100:00080000:0.0:1789762640.574204:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762640.574213:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762640.574249:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504230016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762640.574279:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762640.574282:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762640.574283:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762640.574286:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2567 00000100:00080000:0.0:1789762640.574289:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762640.574291:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762640.574299:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762640.574312:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504230144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762640.574321:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762640.574325:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762640.574326:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762640.574331:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2567 00000100:00080000:0.0:1789762640.574335:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762640.574348:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762640.574357:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762640.574371:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504230272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762640.574385:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762640.574390:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762640.574392:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762640.574398:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2567 00000100:00080000:0.0:1789762640.574402:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762640.574405:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762640.574410:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762640.574422:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504230400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762640.574436:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762640.574441:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789762640.574441:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504230016:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762640.574451:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762640.574468:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2567 00000100:00080000:0.0:1789762640.574473:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762640.574479:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762640.574485:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762640.574507:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504230528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762640.574520:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762640.574525:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762640.574529:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762640.574535:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2567 00000100:00080000:0.0:1789762640.574541:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762640.574546:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762640.574554:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762640.574574:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504230656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762640.574574:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762640.574643:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762640/real 1789762640] req@ffff9018c8b9b100 x1876699504230016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762640.574670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504230016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762640.574692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762640.574702:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762640.574710:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504230016:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762640.574728:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504230144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762640.574811:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762640.574847:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762640/real 1789762640] req@ffff9018c8b9b480 x1876699504230144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762640.574866:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504230144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762640.574894:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762640.574899:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762640.574903:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504230144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762640.574913:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504230272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762640.574972:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762640.574987:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762640/real 1789762640] req@ffff9018c8b98000 x1876699504230272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762640.575002:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504230272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762640.575012:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762640.575016:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762640.575020:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504230272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762640.575116:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504230400:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762640.575145:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762640.575161:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762640/real 1789762640] req@ffff9018c8b9aa00 x1876699504230400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762640.575177:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504230400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762640.575188:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762640.575192:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762640.575196:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504230400:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762640.575203:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504230528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762640.575221:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762640.575233:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762640/real 1789762640] req@ffff9018c8b98e00 x1876699504230528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762640.575247:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504230528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762640.575256:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762640.575260:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762640.575263:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504230528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762640.575271:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504230656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762640.575289:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762640.575300:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762640/real 1789762640] req@ffff9018c8b9a680 x1876699504230656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762640.575331:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504230656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762640.575357:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762640.575363:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762640.575369:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504230656:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762641.214238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762642.238209:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762643.262162:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762644.286211:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762645.310247:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762645.694126:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762645.694138:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762645.694142:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762645.694153:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2572 00000100:00080000:0.0:1789762645.694171:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762645.694180:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762645.694224:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504230784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762645.694254:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762645.694258:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762645.694260:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762645.694264:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2572 00000100:00080000:0.0:1789762645.694268:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762645.694278:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762645.694290:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762645.694297:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504230784:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762645.694303:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504230912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762645.694313:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762645.694316:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762645.694317:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762645.694321:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2572 00000100:00080000:0.0:1789762645.694324:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762645.694328:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762645.694332:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762645.694340:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504231040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762645.694348:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762645.694351:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762645.694352:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762645.694355:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2572 00000100:00080000:0.0:1789762645.694358:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762645.694361:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762645.694366:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762645.694374:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504231168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762645.694374:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762645.694383:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762645.694386:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762645.694387:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762645.694391:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2572 00000100:00080000:0.0:1789762645.694394:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762645.694398:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762645.694402:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762645.694411:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504231296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762645.694418:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762645.694422:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762645.694424:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762645.694424:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762645/real 1789762645] req@ffff9018c8b98380 x1876699504230784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762645.694427:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2572 00000100:00080000:0.0:1789762645.694431:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762645.694434:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762645.694438:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762645.694443:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504230784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762645.694449:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504231424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762645.694456:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762645.694461:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762645.694481:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504230784:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762645.694511:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504230912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762645.694536:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762645.694565:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762645/real 1789762645] req@ffff9018c8b99c00 x1876699504230912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762645.694577:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504230912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762645.694586:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762645.694589:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762645.694593:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504230912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762645.694600:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504231040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762645.694618:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762645.694626:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762645/real 1789762645] req@ffff9018c8b99500 x1876699504231040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762645.694637:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504231040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762645.694644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762645.694646:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762645.694649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504231040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762645.694654:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504231168:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762645.694669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762645.694677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762645/real 1789762645] req@ffff9018c8b9ad80 x1876699504231168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762645.694686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504231168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762645.694694:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762645.694697:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762645.694700:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504231168:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762645.694705:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504231296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762645.694749:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762645.694779:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762645/real 1789762645] req@ffff9018c8b9a300 x1876699504231296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762645.694788:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504231296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762645.694795:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762645.694797:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762645.694799:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504231296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762645.694803:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504231424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762645.694823:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762645.694835:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762645/real 1789762645] req@ffff9018c8b9a680 x1876699504231424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762645.694849:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504231424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762645.694859:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762645.694863:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762645.694866:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504231424:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762646.334339:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762647.358221:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762648.382207:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762649.406189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762650.430250:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762650.814196:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762650.814210:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762650.814214:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762650.814225:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2577 00000100:00080000:1.0:1789762650.814238:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762650.814247:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762650.814302:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504231552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762650.814327:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762650.814330:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762650.814332:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762650.814335:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2577 00000100:00080000:1.0:1789762650.814338:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762650.814342:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762650.814352:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762650.814364:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504231680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762650.814372:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504231552:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762650.814375:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762650.814381:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762650.814383:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762650.814386:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2577 00000100:00080000:1.0:1789762650.814389:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762650.814391:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762650.814394:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762650.814402:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504231808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762650.814408:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762650.814411:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762650.814412:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762650.814415:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2577 00000100:00080000:1.0:1789762650.814417:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762650.814419:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762650.814423:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762650.814429:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504231936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762650.814436:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762650.814439:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762650.814440:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762650.814443:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2577 00000100:00080000:1.0:1789762650.814446:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762650.814450:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762650.814452:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762650.814456:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762650.814464:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504232064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762650.814470:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762650.814473:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762650.814474:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762650.814477:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2577 00000100:00080000:1.0:1789762650.814479:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762650.814482:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762650.814485:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762650.814492:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504232192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762650.814501:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762650/real 1789762650] req@ffff9018c66a5880 x1876699504231552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762650.814518:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504231552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762650.814534:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762650.814537:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762650.814541:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504231552:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762650.814564:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504231680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762650.814580:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762650.814589:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762650/real 1789762650] req@ffff9018c66a7100 x1876699504231680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762650.814598:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504231680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762650.814603:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762650.814606:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762650.814608:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504231680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762650.814612:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504231808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762650.814622:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762650.814628:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762650/real 1789762650] req@ffff9018c66a4a80 x1876699504231808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762650.814635:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504231808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762650.814641:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762650.814643:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762650.814645:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504231808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762650.814649:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504231936:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762650.814660:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762650.814665:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762650/real 1789762650] req@ffff9018c66a7480 x1876699504231936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762650.814672:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504231936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762650.814692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762650.814697:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762650.814699:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504231936:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762650.814704:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504232064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762650.814720:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762650.814727:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762650/real 1789762650] req@ffff9018c66a6a00 x1876699504232064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762650.814735:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504232064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762650.814740:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762650.814743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762650.814745:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504232064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762650.814748:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504232192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762650.814759:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762650.814764:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762650/real 1789762650] req@ffff9018c66a4380 x1876699504232192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762650.814772:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504232192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762650.814777:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762650.814779:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762650.814780:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504232192:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762651.454356:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762652.478167:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762653.502270:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762654.526179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762655.550197:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762655.934135:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762655.934144:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762655.934147:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762655.934155:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2582 00000100:00080000:1.0:1789762655.934164:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762655.934170:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762655.934202:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504232320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762655.934230:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762655.934233:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762655.934234:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762655.934238:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2582 00000100:00080000:1.0:1789762655.934243:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762655.934249:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762655.934260:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762655.934274:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504232320:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762655.934278:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504232448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762655.934297:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762655.934300:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762655.934302:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762655.934306:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2582 00000100:00080000:1.0:1789762655.934309:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762655.934312:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762655.934317:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762655.934332:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504232576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762655.934343:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762655.934347:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762655.934349:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762655.934354:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2582 00000100:00080000:1.0:1789762655.934360:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762655.934367:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762655.934372:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762655.934379:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762655.934391:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504232704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762655.934400:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762655.934402:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762655.934405:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762655.934408:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2582 00000100:00080000:1.0:1789762655.934412:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762655.934424:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762655.934425:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762655/real 1789762655] req@ffff9018c66a4e00 x1876699504232320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762655.934435:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762655.934448:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504232320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762655.934457:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504232832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762655.934463:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762655.934474:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762655.934484:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504232320:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762655.934487:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762655.934499:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762655.934502:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762655.934507:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2582 00000100:00080000:1.0:1789762655.934513:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762655.934516:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762655.934522:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762655.934528:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504232448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762655.934535:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504232960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762655.934559:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762655.934573:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762655/real 1789762655] req@ffff9018c66a5500 x1876699504232448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762655.934587:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504232448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762655.934598:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762655.934600:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762655.934602:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504232448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762655.934607:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504232576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762655.934619:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762655.934625:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762655/real 1789762655] req@ffff9018c66a6680 x1876699504232576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762655.934632:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504232576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762655.934638:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762655.934640:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762655.934653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504232576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762655.934657:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504232704:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762655.934671:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762655.934678:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762655/real 1789762655] req@ffff9018c66a4700 x1876699504232704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762655.934686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504232704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762655.934691:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762655.934693:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762655.934696:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504232704:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762655.934700:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504232832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762655.934711:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762655.934716:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762655/real 1789762655] req@ffff9018c66a6300 x1876699504232832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762655.934723:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504232832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762655.934729:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762655.934730:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762655.934732:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504232832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762655.934745:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504232960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762655.934756:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762655.934761:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762655/real 1789762655] req@ffff9018c66a5180 x1876699504232960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762655.934768:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504232960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762655.934774:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762655.934776:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762655.934778:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504232960:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762656.574187:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762657.598225:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762658.622197:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762659.646181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762660.670174:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762661.054150:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762661.054194:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762661.054198:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762661.054206:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2587 00000100:00080000:2.0:1789762661.054215:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762661.054227:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762661.054261:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504233088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762661.054290:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762661.054293:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762661.054295:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762661.054298:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2587 00000100:00080000:2.0:1789762661.054300:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762661.054303:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762661.054312:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762661.054321:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504233216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762661.054336:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762661.054337:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504233088:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762661.054339:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762661.054343:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762661.054350:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2587 00000100:00080000:2.0:1789762661.054352:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762661.054354:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762661.054358:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762661.054365:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504233344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762661.054376:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762661.054378:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762661.054380:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762661.054383:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2587 00000100:00080000:2.0:1789762661.054385:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762661.054387:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762661.054390:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762661.054398:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504233472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762661.054404:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762661.054406:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762661.054408:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762661.054410:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2587 00000100:00080000:2.0:1789762661.054413:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762661.054415:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762661.054419:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762661.054426:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504233600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762661.054432:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762661.054435:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762661.054436:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762661.054440:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762661.054444:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2587 00000100:00080000:2.0:1789762661.054448:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762661.054451:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762661.054455:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762661.054461:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504233728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762661.054487:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762661/real 1789762661] req@ffff9018d0f24380 x1876699504233088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762661.054507:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504233088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762661.054519:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762661.054523:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762661.054528:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504233088:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762661.054550:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504233216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762661.054576:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762661.054591:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762661/real 1789762661] req@ffff9018d0f27800 x1876699504233216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762661.054618:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504233216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762661.054627:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762661.054631:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762661.054635:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504233216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762661.054642:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504233344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762661.054663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762661.054676:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762661/real 1789762661] req@ffff9018d0f25180 x1876699504233344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762661.054689:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504233344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762661.054697:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762661.054700:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762661.054705:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504233344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762661.054711:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504233472:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762661.054733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762661.054746:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762661/real 1789762661] req@ffff9018d0f25c00 x1876699504233472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762661.054759:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504233472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762661.054768:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762661.054771:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762661.054775:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504233472:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762661.054782:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504233600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762661.054803:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762661.054816:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762661/real 1789762661] req@ffff9018d0f24000 x1876699504233600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762661.054827:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504233600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762661.054836:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762661.054840:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762661.054843:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504233600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762661.054849:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504233728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762661.054872:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762661.054884:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762661/real 1789762661] req@ffff9018d0f24e00 x1876699504233728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762661.054896:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504233728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762661.054905:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762661.054909:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762661.054912:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504233728:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762661.694202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762662.718160:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762663.742152:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762664.766169:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762665.790158:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762666.174141:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762666.174149:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762666.174152:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762666.174160:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2592 00000100:00080000:0.0:1789762666.174169:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762666.174174:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762666.174205:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504233856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762666.174230:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762666.174232:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762666.174234:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762666.174236:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2592 00000100:00080000:0.0:1789762666.174239:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762666.174241:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762666.174249:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762666.174259:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504233984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762666.174267:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762666.174270:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762666.174273:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762666.174275:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504233856:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762666.174277:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2592 00000100:00080000:0.0:1789762666.174283:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762666.174286:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762666.174289:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762666.174297:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504234112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762666.174303:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762666.174305:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762666.174307:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762666.174309:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2592 00000100:00080000:0.0:1789762666.174312:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762666.174314:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762666.174317:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762666.174324:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504234240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762666.174334:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762666.174338:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762666.174342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762666.174345:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762666.174351:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2592 00000100:00080000:0.0:1789762666.174355:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762666.174360:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762666.174366:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762666.174376:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762666/real 1789762666] req@ffff9018c8b98e00 x1876699504233856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762666.174382:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504234368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762666.174398:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762666.174402:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504233856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762666.174403:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762666.174407:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762666.174414:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2592 00000100:00080000:0.0:1789762666.174421:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762666.174423:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762666.174427:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789762666.174431:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:0.0:1789762666.174434:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762666.174437:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504233856:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762666.174446:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504234496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762666.174461:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504233984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762666.174479:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762666.174487:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762666/real 1789762666] req@ffff9018c8b9aa00 x1876699504233984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762666.174495:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504233984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762666.174500:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762666.174502:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762666.174505:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504233984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762666.174508:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504234112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762666.174518:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762666.174524:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762666/real 1789762666] req@ffff9018c8b98000 x1876699504234112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762666.174531:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504234112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762666.174536:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762666.174538:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762666.174540:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504234112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762666.174543:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504234240:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762666.174553:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762666.174559:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762666/real 1789762666] req@ffff9018c8b9b480 x1876699504234240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762666.174579:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504234240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762666.174584:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762666.174586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762666.174588:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504234240:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762666.174592:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504234368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762666.174604:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762666.174610:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762666/real 1789762666] req@ffff9018c8b9b100 x1876699504234368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762666.174618:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504234368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762666.174622:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762666.174624:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762666.174626:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504234368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762666.174629:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504234496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762666.174638:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762666.174644:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762666/real 1789762666] req@ffff9018c8b9b800 x1876699504234496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762666.174651:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504234496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762666.174656:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762666.174658:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762666.174660:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504234496:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762666.814238:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762667.838130:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762668.862176:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762669.886195:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762670.910252:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762671.294131:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762671.294139:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762671.294142:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762671.294151:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2597 00000100:00080000:2.0:1789762671.294161:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762671.294168:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762671.294206:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504234624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762671.294231:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762671.294234:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762671.294235:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762671.294239:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2597 00000100:00080000:2.0:1789762671.294242:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762671.294244:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762671.294252:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762671.294265:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504234752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762671.294274:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762671.294277:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762671.294278:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762671.294281:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2597 00000100:00080000:2.0:1789762671.294283:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762671.294295:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762671.294296:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504234624:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789762671.294305:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762671.294316:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504234880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762671.294324:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762671.294326:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762671.294328:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762671.294331:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2597 00000100:00080000:2.0:1789762671.294334:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762671.294337:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762671.294341:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762671.294349:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504235008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762671.294357:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762671.294360:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762671.294361:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762671.294363:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2597 00000100:00080000:2.0:1789762671.294365:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762671.294369:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762671.294373:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762671.294381:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504235136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762671.294390:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762671.294403:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762671.294406:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762671.294407:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762671.294413:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2597 00000100:00080000:2.0:1789762671.294423:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762671.294426:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762671.294429:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762671.294444:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504235264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762671.294453:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762671/real 1789762671] req@ffff9018d0f24700 x1876699504234624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762671.294473:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504234624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762671.294484:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762671.294488:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762671.294491:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504234624:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762671.294516:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504234752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762671.294552:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762671.294563:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762671/real 1789762671] req@ffff9018d0f26680 x1876699504234752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762671.294578:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504234752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762671.294584:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762671.294587:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762671.294589:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504234752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762671.294594:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504234880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762671.294608:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762671.294615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762671/real 1789762671] req@ffff9018d0f27480 x1876699504234880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762671.294623:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504234880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762671.294628:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762671.294629:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762671.294631:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504234880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762671.294635:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504235008:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762671.294647:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762671.294653:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762671/real 1789762671] req@ffff9018d0f25500 x1876699504235008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762671.294660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504235008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762671.294666:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762671.294668:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762671.294669:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504235008:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762671.294673:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504235136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762671.294684:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762671.294689:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762671/real 1789762671] req@ffff9018d0f24e00 x1876699504235136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762671.294697:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504235136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762671.294702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762671.294704:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762671.294706:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504235136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762671.294709:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504235264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762671.294720:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762671.294726:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762671/real 1789762671] req@ffff9018d0f24000 x1876699504235264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762671.294734:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504235264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762671.294739:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762671.294740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762671.294742:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504235264:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762671.934242:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762672.958170:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762673.982238:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762675.006193:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762676.030178:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762676.414121:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762676.414133:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762676.414138:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762676.414149:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2602 00000100:00080000:2.0:1789762676.414160:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762676.414169:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762676.414205:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504235392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762676.414236:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762676.414241:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762676.414243:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762676.414249:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2602 00000100:00080000:2.0:1789762676.414255:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762676.414260:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762676.414271:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762676.414272:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504235392:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762676.414289:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504235520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762676.414300:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762676.414303:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762676.414305:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762676.414308:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2602 00000100:00080000:2.0:1789762676.414311:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762676.414315:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762676.414319:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762676.414328:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504235648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762676.414340:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762676.414341:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762676.414346:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762676.414349:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762676.414355:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2602 00000100:00080000:2.0:1789762676.414360:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762676.414366:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762676.414372:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762676.414378:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762676/real 1789762676] req@ffff9018d0f25c00 x1876699504235392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762676.414388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504235776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762676.414401:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504235392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762676.414419:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762676.414421:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762676.414423:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762676.414427:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762676.414428:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789762676.414432:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2602 00000100:00100000:0.0:1789762676.414435:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504235392:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762676.414437:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762676.414442:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762676.414447:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762676.414456:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504235904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762676.414460:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504235520:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789762676.414470:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762676.414472:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762676.414474:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762676.414477:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2602 00000100:00080000:2.0:1789762676.414480:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762676.414483:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762676.414486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762676.414505:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762676.414516:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504236032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762676.414523:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762676/real 1789762676] req@ffff9018d0f25180 x1876699504235520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762676.414536:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504235520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762676.414544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762676.414548:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762676.414551:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504235520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762676.414557:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504235648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762676.414574:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762676.414583:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762676/real 1789762676] req@ffff9018d0f27800 x1876699504235648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762676.414594:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504235648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762676.414600:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762676.414603:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762676.414606:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504235648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762676.414610:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504235776:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762676.414632:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762676.414639:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762676/real 1789762676] req@ffff9018d0f24380 x1876699504235776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762676.414647:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504235776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762676.414654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762676.414656:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762676.414658:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504235776:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762676.414670:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504235904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762676.414682:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762676.414689:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762676/real 1789762676] req@ffff9018d0f26a00 x1876699504235904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762676.414698:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504235904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762676.414704:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762676.414707:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762676.414710:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504235904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762676.414716:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504236032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762676.414728:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762676.414737:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762676/real 1789762676] req@ffff9018d0f24a80 x1876699504236032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762676.414751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504236032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762676.414758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762676.414762:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762676.414766:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504236032:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762677.054178:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762678.078193:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762679.102255:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762680.126230:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762681.150162:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762681.534100:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762681.534108:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762681.534111:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762681.534120:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2608 00000100:00080000:2.0:1789762681.534129:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762681.534135:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762681.534164:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504236160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762681.534186:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762681.534189:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762681.534190:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762681.534193:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2608 00000100:00080000:2.0:1789762681.534195:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762681.534197:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762681.534205:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762681.534214:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504236288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762681.534221:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762681.534224:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762681.534225:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762681.534228:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2608 00000100:00080000:2.0:1789762681.534230:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762681.534235:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762681.534236:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504236160:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789762681.534238:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762681.534250:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504236416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762681.534257:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762681.534260:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762681.534261:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762681.534264:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2608 00000100:00080000:2.0:1789762681.534267:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762681.534269:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762681.534272:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762681.534279:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504236544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762681.534286:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762681.534288:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762681.534290:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762681.534295:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2608 00000800:00000100:0.0:1789762681.534297:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762681.534299:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762681.534304:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762681.534307:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762681.534315:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504236672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762681.534321:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762681.534324:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762681.534326:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762681.534329:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2608 00000100:00100000:0.0:1789762681.534334:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762681/real 1789762681] req@ffff9018d0f26d80 x1876699504236160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762681.534335:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762681.534343:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762681.534352:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762681.534355:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504236160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762681.534365:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789762681.534369:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504236800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762681.534372:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762681.534381:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504236160:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762681.534393:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504236288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762681.534419:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762681.534432:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762681/real 1789762681] req@ffff9018d0f25f80 x1876699504236288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762681.534442:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504236288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762681.534447:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762681.534465:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762681.534469:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504236288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762681.534488:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504236416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762681.534503:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762681.534511:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762681/real 1789762681] req@ffff9018d0f27100 x1876699504236416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762681.534518:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504236416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762681.534524:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762681.534526:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762681.534528:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504236416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762681.534531:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504236544:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762681.534541:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762681.534546:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762681/real 1789762681] req@ffff9018d0f26300 x1876699504236544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762681.534553:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504236544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762681.534559:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762681.534561:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762681.534562:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504236544:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762681.534566:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504236672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762681.534574:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762681.534580:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762681/real 1789762681] req@ffff9018d0f24a80 x1876699504236672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762681.534588:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504236672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762681.534593:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762681.534594:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762681.534596:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504236672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762681.534599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504236800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762681.534609:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762681.534615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762681/real 1789762681] req@ffff9018d0f26a00 x1876699504236800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762681.534622:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504236800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762681.534627:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762681.534628:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762681.534630:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504236800:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762682.174167:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762683.198144:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762684.222268:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762685.246245:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762686.270218:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762686.654191:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762686.654202:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762686.654206:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762686.654216:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2613 00000100:00080000:1.0:1789762686.654229:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762686.654236:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762686.654279:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504236928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762686.654308:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762686.654312:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762686.654313:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762686.654316:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2613 00000100:00080000:1.0:1789762686.654319:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762686.654321:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762686.654331:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762686.654339:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504237056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762686.654347:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762686.654350:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762686.654351:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762686.654354:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2613 00000100:00080000:1.0:1789762686.654356:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762686.654358:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762686.654360:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762686.654367:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504237184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762686.654375:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504236928:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762686.654375:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762686.654381:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762686.654384:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762686.654389:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2613 00000100:00080000:1.0:1789762686.654391:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762686.654393:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762686.654397:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762686.654405:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504237312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762686.654421:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762686.654426:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762686.654427:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762686.654430:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2613 00000100:00080000:1.0:1789762686.654433:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762686.654437:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762686.654441:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762686.654450:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504237440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762686.654456:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762686.654459:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762686.654461:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762686.654463:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2613 00000100:00080000:1.0:1789762686.654466:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762686.654467:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762686.654470:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762686.654479:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762686.654485:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504237568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762686.654532:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762686/real 1789762686] req@ffff9018c66a7800 x1876699504236928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762686.654561:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504236928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762686.654574:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762686.654578:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762686.654583:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504236928:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762686.654592:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504237056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762686.654613:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762686.654626:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762686/real 1789762686] req@ffff9018c66a7b80 x1876699504237056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762686.654639:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504237056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762686.654648:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762686.654651:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762686.654655:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504237056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762686.654681:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504237184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762686.654700:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762686.654711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762686/real 1789762686] req@ffff9018c66a4000 x1876699504237184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762686.654722:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504237184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762686.654738:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762686.654741:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762686.654745:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504237184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762686.654751:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504237312:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762686.654768:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762686.654777:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762686/real 1789762686] req@ffff9018c66a5f80 x1876699504237312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762686.654789:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504237312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762686.654797:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762686.654801:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762686.654804:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504237312:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762686.654809:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504237440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762686.654825:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762686.654835:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762686/real 1789762686] req@ffff9018c66a6d80 x1876699504237440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762686.654847:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504237440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762686.654856:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762686.654859:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762686.654863:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504237440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762686.654869:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504237568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762686.654887:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762686.654897:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762686/real 1789762686] req@ffff9018c66a5c00 x1876699504237568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762686.654909:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504237568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762686.654917:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762686.654920:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762686.654924:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504237568:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762687.294215:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762688.318254:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762689.342129:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762690.366197:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762691.390143:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762691.774138:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762691.774149:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762691.774153:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762691.774163:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2618 00000100:00080000:2.0:1789762691.774174:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762691.774181:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762691.774218:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504237696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762691.774247:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762691.774256:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762691.774259:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762691.774263:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2618 00000100:00080000:2.0:1789762691.774268:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762691.774273:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762691.774283:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762691.774290:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504237696:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762691.774301:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504237824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762691.774315:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762691.774319:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762691.774321:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762691.774326:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2618 00000100:00080000:2.0:1789762691.774331:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762691.774335:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762691.774341:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762691.774354:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504237952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762691.774365:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762691.774369:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762691.774371:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762691.774390:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2618 00000100:00080000:2.0:1789762691.774394:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762691.774397:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762691.774403:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762691.774410:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504237824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762691.774419:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504238080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762691.774434:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762691.774437:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762691.774440:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762691.774447:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504237952:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789762691.774448:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2618 00000100:00080000:2.0:1789762691.774457:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762691.774463:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762691.774470:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762691.774481:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504238080:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762691.774486:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504238208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762691.774511:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762691.774515:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762691.774518:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762691.774525:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2618 00000100:00100000:0.0:1789762691.774529:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504238208:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789762691.774529:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762691.774538:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762691.774544:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762691.774559:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504238336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762691.774611:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504238336:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762692.414269:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762693.438209:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762694.462248:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762694.846407:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789762694.846457:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762694.846476:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762694.846520:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762694.846530:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762694.846542:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762694.846553:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789762694.846567:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762691/real 1789762694] req@ffff9018d0f25c00 x1876699504238080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1789762694.846571:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789762694.846601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504238080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762694.846615:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762694.846619:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762694.846635:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504238080:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762694.846643:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762691/real 1789762694] req@ffff9018d0f27800 x1876699504237824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762694.846661:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504237824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762694.846670:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762694.846673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762694.846677:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504237824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762694.846684:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762691/real 1789762694] req@ffff9018d0f25180 x1876699504237952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762694.846697:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504237952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762694.846705:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762694.846709:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762694.846712:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504237952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762694.846719:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762691/real 1789762694] req@ffff9018d0f24380 x1876699504237696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762694.846731:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504237696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762694.846738:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762694.846741:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762694.846744:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504237696:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762694.846768:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762691/real 1789762694] req@ffff9018d0f24e00 x1876699504238336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762694.846781:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504238336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762694.846789:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762694.846793:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762694.846796:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504238336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762694.846802:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762691/real 1789762694] req@ffff9018d0f24000 x1876699504238208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762694.846815:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504238208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762694.846823:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762694.846826:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762694.846829:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504238208:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762695.486209:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762696.510234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762696.894091:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762696.894099:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762696.894103:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762696.894112:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2623 00000100:00080000:2.0:1789762696.894121:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762696.894126:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762696.894157:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504238464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762696.894182:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762696.894186:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762696.894189:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762696.894193:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2623 00000100:00080000:2.0:1789762696.894197:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762696.894201:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762696.894210:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762696.894224:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504238592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762696.894231:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504238464:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762696.894234:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762696.894237:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762696.894239:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762696.894242:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2623 00000100:00080000:2.0:1789762696.894245:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762696.894248:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762696.894252:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762696.894261:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504238720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762696.894269:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762696.894272:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762696.894274:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762696.894277:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2623 00000100:00080000:2.0:1789762696.894288:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762696.894291:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789762696.894295:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789762696.894296:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762696.894305:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504238848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762696.894314:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762696.894317:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762696.894319:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762696.894322:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2623 00000100:00080000:2.0:1789762696.894325:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762696.894329:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762696.894331:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762696/real 1789762696] req@ffff9018d0f25500 x1876699504238464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789762696.894333:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762696.894361:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504238976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762696.894364:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504238464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762696.894372:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762696.894375:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762696.894375:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762696.894377:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762696.894379:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789762696.894380:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2623 00000100:00100000:0.0:1789762696.894383:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504238464:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762696.894384:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762696.894387:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762696.894396:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762696.894406:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504239104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762696.894408:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504238592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762696.894431:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762696.894446:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762696/real 1789762696] req@ffff9018d0f27480 x1876699504238592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762696.894459:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504238592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762696.894468:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762696.894472:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762696.894475:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504238592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762696.894482:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504238720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762696.894503:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762696.894515:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762696/real 1789762696] req@ffff9018d0f26680 x1876699504238720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762696.894533:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504238720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762696.894540:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762696.894543:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762696.894546:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504238720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762696.894552:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504238848:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762696.894579:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762696.894591:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762696/real 1789762696] req@ffff9018d0f24700 x1876699504238848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762696.894602:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504238848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762696.894609:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762696.894612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762696.894615:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504238848:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762696.894620:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504238976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762696.894640:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762696.894652:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762696/real 1789762696] req@ffff9018d0f24000 x1876699504238976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762696.894662:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504238976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762696.894675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762696.894679:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762696.894682:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504238976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762696.894701:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504239104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762696.894723:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762696.894735:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762696/real 1789762696] req@ffff9018d0f24e00 x1876699504239104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762696.894748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504239104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762696.894756:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762696.894759:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762696.894763:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504239104:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762697.534241:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762698.558261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762699.582256:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762700.606249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762701.630281:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762702.014202:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762702.014212:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762702.014217:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762702.014228:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2628 00000100:00080000:3.0:1789762702.014241:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762702.014248:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762702.014309:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504239232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762702.014341:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762702.014344:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762702.014345:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762702.014348:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2628 00000100:00080000:3.0:1789762702.014351:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762702.014354:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762702.014363:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762702.014373:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504239360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762702.014381:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762702.014384:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762702.014386:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762702.014389:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2628 00000100:00080000:3.0:1789762702.014392:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762702.014395:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762702.014399:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762702.014405:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504239232:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762702.014407:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504239488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762702.014417:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762702.014420:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762702.014421:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762702.014424:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2628 00000100:00080000:3.0:1789762702.014425:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762702.014428:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762702.014431:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762702.014439:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504239616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762702.014445:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762702.014448:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762702.014449:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762702.014451:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2628 00000100:00080000:3.0:1789762702.014455:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762702.014458:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762702.014461:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762702.014467:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504239744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762702.014479:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762702.014483:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1789762702.014485:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762702.014486:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762702.014493:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2628 00000100:00080000:3.0:1789762702.014495:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762702.014498:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762702.014501:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762702.014508:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504239872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762702.014546:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762702/real 1789762702] req@ffff9018d070ce00 x1876699504239232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762702.014569:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504239232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762702.014580:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762702.014585:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762702.014590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504239232:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762702.014598:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504239360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762702.014625:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762702.014640:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762702/real 1789762702] req@ffff9018d070ca80 x1876699504239360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762702.014653:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504239360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762702.014662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762702.014665:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762702.014669:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504239360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762702.014710:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504239488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762702.014728:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762702.014736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762702/real 1789762702] req@ffff9018d070fb80 x1876699504239488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762702.014744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504239488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762702.014750:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762702.014752:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762702.014754:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504239488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762702.014758:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504239616:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762702.014769:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762702.014775:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762702/real 1789762702] req@ffff9018d070f480 x1876699504239616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762702.014783:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504239616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762702.014788:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762702.014790:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762702.014791:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504239616:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762702.014795:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504239744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762702.014805:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762702.014811:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762702/real 1789762702] req@ffff9018d070ed80 x1876699504239744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762702.014818:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504239744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762702.014823:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762702.014825:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762702.014827:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504239744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762702.014830:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504239872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762702.014839:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762702.014845:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762702/real 1789762702] req@ffff9018d070df80 x1876699504239872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762702.014857:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504239872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762702.014862:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762702.014864:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762702.014866:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504239872:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762702.654204:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762703.678159:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762704.702227:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762705.726197:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762706.750197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762707.134159:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762707.134169:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762707.134174:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762707.134185:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2633 00000100:00080000:1.0:1789762707.134198:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762707.134205:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762707.134252:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504240000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762707.134294:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762707.134298:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762707.134299:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762707.134303:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2633 00000100:00080000:1.0:1789762707.134306:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762707.134309:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762707.134319:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762707.134330:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504240128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762707.134338:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762707.134340:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762707.134342:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762707.134345:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2633 00000100:00080000:1.0:1789762707.134348:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762707.134350:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762707.134355:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762707.134361:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504240256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762707.134368:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762707.134370:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762707.134374:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762707.134376:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504240000:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762707.134379:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2633 00000100:00080000:1.0:1789762707.134385:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762707.134387:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762707.134391:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762707.134399:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504240384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762707.134406:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762707.134409:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762707.134410:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762707.134413:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2633 00000100:00080000:1.0:1789762707.134415:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762707.134418:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762707.134421:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762707.134428:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504240512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762707.134433:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762707.134436:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762707.134438:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762707.134441:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2633 00000100:00080000:1.0:1789762707.134444:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762707.134446:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762707.134450:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762707.134454:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762707.134462:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504240640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762707.134507:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762707/real 1789762707] req@ffff9018c66a5c00 x1876699504240000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762707.134522:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504240000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762707.134536:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762707.134540:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762707.134543:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504240000:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762707.134549:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504240128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762707.134565:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762707.134573:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762707/real 1789762707] req@ffff9018c66a6d80 x1876699504240128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762707.134581:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504240128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762707.134588:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762707.134590:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762707.134592:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504240128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762707.134616:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504240256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762707.134626:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762707.134632:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762707/real 1789762707] req@ffff9018c66a5f80 x1876699504240256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762707.134640:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504240256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762707.134645:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762707.134647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762707.134649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504240256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762707.134652:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504240384:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762707.134663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762707.134669:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762707/real 1789762707] req@ffff9018c66a4000 x1876699504240384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762707.134676:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504240384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762707.134682:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762707.134685:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762707.134689:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504240384:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762707.134695:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504240512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762707.134714:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762707.134726:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762707/real 1789762707] req@ffff9018c66a7b80 x1876699504240512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762707.134745:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504240512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762707.134751:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762707.134753:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762707.134755:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504240512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762707.134759:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504240640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762707.134774:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762707.134781:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762707/real 1789762707] req@ffff9018c66a7800 x1876699504240640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762707.134789:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504240640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762707.134794:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762707.134796:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762707.134798:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504240640:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762707.774289:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762708.798211:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762709.822311:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762710.846170:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762711.870217:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762712.254163:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762712.254173:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762712.254177:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762712.254191:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2638 00000100:00080000:1.0:1789762712.254204:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762712.254212:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762712.254275:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504240768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762712.254301:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762712.254304:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762712.254306:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762712.254311:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2638 00000100:00080000:1.0:1789762712.254315:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762712.254319:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762712.254331:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762712.254347:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504240896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762712.254364:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504240768:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762712.254369:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762712.254377:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762712.254380:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762712.254387:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2638 00000100:00080000:1.0:1789762712.254395:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762712.254399:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762712.254405:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762712.254418:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504241024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762712.254426:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762712.254429:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762712.254430:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762712.254435:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2638 00000100:00080000:1.0:1789762712.254441:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762712.254446:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762712.254451:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789762712.254461:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762712.254463:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504241152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762712.254477:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762712.254481:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762712.254482:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762712.254486:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2638 00000100:00080000:1.0:1789762712.254490:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762712.254493:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762712.254495:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762712.254504:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504241280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762712.254511:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762712.254513:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762712.254514:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762712.254517:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2638 00000100:00080000:1.0:1789762712.254520:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762712.254525:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762712.254526:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762712/real 1789762712] req@ffff9018c66a5180 x1876699504240768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762712.254529:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762712.254542:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504241408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762712.254564:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504240768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762712.254579:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762712.254585:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762712.254590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504240768:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762712.254629:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504240896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762712.254663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762712.254698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762712/real 1789762712] req@ffff9018c66a6300 x1876699504240896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762712.254713:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504240896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762712.254723:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762712.254731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762712.254734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504240896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762712.254741:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504241024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762712.254761:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762712.254772:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762712/real 1789762712] req@ffff9018c66a4700 x1876699504241024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762712.254791:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504241024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762712.254800:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762712.254803:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762712.254807:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504241024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762712.254813:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504241152:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762712.254830:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762712.254840:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762712/real 1789762712] req@ffff9018c66a6680 x1876699504241152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762712.254852:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504241152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762712.254861:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762712.254864:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762712.254868:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504241152:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762712.254873:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504241280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762712.254890:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762712.254899:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762712/real 1789762712] req@ffff9018c66a5500 x1876699504241280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762712.254911:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504241280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762712.254926:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762712.254929:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762712.254932:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504241280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762712.254938:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504241408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762712.254955:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762712.254965:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762712/real 1789762712] req@ffff9018c66a4e00 x1876699504241408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762712.254977:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504241408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762712.254986:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762712.254990:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762712.254993:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504241408:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762712.894243:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762713.918261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762714.942160:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762715.966209:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762716.990265:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762717.374233:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762717.374242:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762717.374246:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762717.374255:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2643 00000100:00080000:2.0:1789762717.374263:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762717.374269:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762717.374301:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504241536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762717.374329:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762717.374332:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762717.374333:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762717.374336:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2643 00000100:00080000:2.0:1789762717.374339:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762717.374341:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762717.374350:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762717.374358:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504241664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762717.374366:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762717.374368:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762717.374369:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762717.374372:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2643 00000100:00080000:2.0:1789762717.374374:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762717.374377:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762717.374382:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504241536:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789762717.374383:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762717.374391:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504241792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762717.374403:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762717.374405:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762717.374407:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762717.374409:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2643 00000100:00080000:2.0:1789762717.374411:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762717.374413:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762717.374416:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762717.374425:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504241920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762717.374432:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762717.374435:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762717.374436:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762717.374438:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2643 00000100:00080000:2.0:1789762717.374440:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762717.374443:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762717.374447:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762717.374455:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504242048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762717.374463:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762717.374464:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762717.374469:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762717.374471:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762717.374474:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2643 00000100:00080000:2.0:1789762717.374484:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762717.374486:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762717.374490:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762717.374497:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504242176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762717.374522:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762717/real 1789762717] req@ffff9018d0f24380 x1876699504241536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762717.374539:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504241536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762717.374547:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762717.374551:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762717.374553:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504241536:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762717.374559:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504241664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762717.374578:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762717.374587:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762717/real 1789762717] req@ffff9018d0f25180 x1876699504241664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762717.374595:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504241664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762717.374601:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762717.374603:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762717.374605:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504241664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762717.374623:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504241792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762717.374634:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762717.374640:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762717/real 1789762717] req@ffff9018d0f27800 x1876699504241792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762717.374655:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504241792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762717.374660:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762717.374662:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762717.374676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504241792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762717.374680:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504241920:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762717.374692:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762717.374698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762717/real 1789762717] req@ffff9018d0f25c00 x1876699504241920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762717.374706:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504241920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762717.374712:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762717.374713:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762717.374715:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504241920:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762717.374719:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504242048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762717.374728:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762717.374734:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762717/real 1789762717] req@ffff9018d0f26a00 x1876699504242048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762717.374741:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504242048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762717.374746:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762717.374748:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762717.374750:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504242048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762717.374753:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504242176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762717.374770:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762717.374782:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762717/real 1789762717] req@ffff9018d0f24a80 x1876699504242176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762717.374802:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504242176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762717.374811:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762717.374813:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762717.374815:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504242176:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762718.014265:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762719.038308:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762720.062294:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762721.086179:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762722.110206:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762722.494109:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762722.494117:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762722.494120:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762722.494129:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2649 00000100:00080000:2.0:1789762722.494138:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762722.494158:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762722.494191:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504242304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762722.494213:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762722.494216:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762722.494218:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762722.494220:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2649 00000100:00080000:2.0:1789762722.494223:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762722.494225:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762722.494233:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762722.494241:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504242432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762722.494254:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762722.494256:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762722.494258:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762722.494260:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2649 00000100:00080000:2.0:1789762722.494262:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762722.494266:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504242304:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762722.494267:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762722.494272:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762722.494279:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504242560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762722.494285:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762722.494288:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762722.494289:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762722.494291:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2649 00000100:00080000:2.0:1789762722.494293:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762722.494296:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762722.494299:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762722.494307:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504242688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762722.494313:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762722.494316:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762722.494318:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762722.494320:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2649 00000100:00080000:2.0:1789762722.494323:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762722.494326:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789762722.494330:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789762722.494333:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762722.494343:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504242816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762722.494350:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762722.494352:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762722.494354:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762722.494356:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2649 00000100:00080000:2.0:1789762722.494359:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762722.494361:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762722.494365:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762722.494369:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762722/real 1789762722] req@ffff9018d0f26300 x1876699504242304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762722.494375:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504242944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762722.494390:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504242304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762722.494399:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762722.494403:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762722.494406:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504242304:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762722.494411:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504242432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762722.494426:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762722.494443:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762722/real 1789762722] req@ffff9018d0f27100 x1876699504242432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762722.494452:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504242432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762722.494458:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762722.494460:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762722.494462:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504242432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762722.494481:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504242560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762722.494493:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762722.494500:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762722/real 1789762722] req@ffff9018d0f25f80 x1876699504242560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762722.494507:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504242560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762722.494512:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762722.494514:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762722.494516:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504242560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762722.494519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504242688:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762722.494530:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762722.494536:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762722/real 1789762722] req@ffff9018d0f26d80 x1876699504242688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762722.494543:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504242688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762722.494548:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762722.494550:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762722.494552:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504242688:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762722.494556:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504242816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762722.494566:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762722.494571:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762722/real 1789762722] req@ffff9018d0f24a80 x1876699504242816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762722.494578:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504242816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762722.494583:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762722.494585:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762722.494587:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504242816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762722.494590:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504242944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762722.494599:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762722.494604:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762722/real 1789762722] req@ffff9018d0f26a00 x1876699504242944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762722.494611:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504242944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762722.494616:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762722.494618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762722.494619:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504242944:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762723.134172:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762724.158150:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762725.182216:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762726.206176:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762727.230271:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762727.614154:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762727.614162:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762727.614165:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762727.614174:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2654 00000100:00080000:2.0:1789762727.614183:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762727.614189:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762727.614223:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504243072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762727.614245:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762727.614248:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762727.614250:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762727.614253:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2654 00000100:00080000:2.0:1789762727.614255:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762727.614257:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762727.614266:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762727.614274:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504243200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762727.614281:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762727.614284:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762727.614286:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762727.614290:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2654 00000100:00080000:2.0:1789762727.614292:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762727.614295:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762727.614298:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762727.614304:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504243328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762727.614312:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762727.614315:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762727.614316:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762727.614318:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2654 00000100:00080000:2.0:1789762727.614320:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762727.614325:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762727.614330:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762727.614336:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504243072:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762727.614339:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504243456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762727.614359:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762727.614362:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762727.614363:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762727.614367:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2654 00000100:00080000:2.0:1789762727.614369:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762727.614372:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762727.614376:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762727.614384:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504243584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762727.614390:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762727.614392:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762727.614394:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762727.614397:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2654 00000100:00080000:2.0:1789762727.614404:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762727.614408:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762727.614411:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762727.614418:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504243712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762727.614435:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762727.614483:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762727/real 1789762727] req@ffff9018d0f25c00 x1876699504243072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762727.614498:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504243072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762727.614508:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762727.614511:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762727.614514:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504243072:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762727.614521:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504243200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762727.614537:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762727.614545:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762727/real 1789762727] req@ffff9018d0f27800 x1876699504243200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762727.614553:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504243200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762727.614558:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762727.614560:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762727.614562:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504243200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762727.614597:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504243328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762727.614610:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762727.614616:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762727/real 1789762727] req@ffff9018d0f25180 x1876699504243328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762727.614629:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504243328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762727.614635:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762727.614637:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762727.614638:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504243328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762727.614642:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504243456:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762727.614654:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762727.614660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762727/real 1789762727] req@ffff9018d0f24380 x1876699504243456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762727.614667:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504243456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762727.614673:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762727.614674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762727.614676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504243456:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762727.614680:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504243584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762727.614690:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762727.614696:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762727/real 1789762727] req@ffff9018d0f24e00 x1876699504243584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762727.614703:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504243584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762727.614713:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762727.614716:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762727.614718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504243584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762727.614721:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504243712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762727.614731:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762727.614737:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762727/real 1789762727] req@ffff9018d0f24000 x1876699504243712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762727.614743:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504243712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762727.614748:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762727.614750:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762727.614753:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504243712:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762728.254183:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762729.278248:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762730.302234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762731.326194:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762732.350189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762732.734249:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762732.734257:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762732.734261:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762732.734272:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2659 00000100:00080000:1.0:1789762732.734281:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762732.734288:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762732.734324:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504243840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762732.734352:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762732.734357:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762732.734359:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762732.734363:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2659 00000100:00080000:1.0:1789762732.734368:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762732.734373:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762732.734385:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762732.734399:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504243968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762732.734408:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504243840:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762732.734412:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762732.734417:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762732.734419:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762732.734423:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2659 00000100:00080000:1.0:1789762732.734425:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762732.734428:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762732.734431:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762732.734444:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504244096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762732.734454:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762732.734458:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762732.734460:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762732.734464:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2659 00000100:00080000:1.0:1789762732.734469:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762732.734473:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789762732.734478:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:1.0:1789762732.734480:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762732.734498:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504244224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762732.734507:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762732.734510:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762732.734511:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762732.734515:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2659 00000100:00100000:0.0:1789762732.734519:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762732/real 1789762732] req@ffff9018c66a4380 x1876699504243840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762732.734519:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762732.734535:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762732.734541:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762732.734547:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504243840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762732.734553:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504244352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762732.734559:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762732.734563:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762732.734567:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789762732.734568:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762732.734572:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762732.734574:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504243840:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762732.734578:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2659 00000100:00080000:1.0:1789762732.734582:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762732.734585:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762732.734589:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762732.734596:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504244480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762732.734600:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504243968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762732.734626:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762732.734637:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762732/real 1789762732] req@ffff9018c66a6a00 x1876699504243968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762732.734645:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504243968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762732.734651:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762732.734654:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762732.734656:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504243968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762732.734660:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504244096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762732.734678:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762732.734684:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762732/real 1789762732] req@ffff9018c66a7480 x1876699504244096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762732.734691:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504244096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762732.734697:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762732.734698:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762732.734701:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504244096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762732.734704:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504244224:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762732.734716:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762732.734721:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762732/real 1789762732] req@ffff9018c66a4a80 x1876699504244224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762732.734735:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504244224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762732.734741:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762732.734743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762732.734745:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504244224:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762732.734749:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504244352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762732.734760:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762732.734766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762732/real 1789762732] req@ffff9018c66a7100 x1876699504244352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762732.734773:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504244352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762732.734778:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762732.734780:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762732.734782:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504244352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762732.734798:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504244480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762732.734810:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762732.734817:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762732/real 1789762732] req@ffff9018c66a5880 x1876699504244480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762732.734829:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504244480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762732.734834:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762732.734836:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762732.734838:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504244480:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762733.374226:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762734.398174:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762735.422152:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762736.446181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762737.470267:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762737.854184:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762737.854202:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762737.854208:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762737.854224:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2664 00000100:00080000:3.0:1789762737.854241:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762737.854250:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762737.854322:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504244608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762737.854357:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762737.854362:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762737.854364:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762737.854369:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2664 00000100:00080000:3.0:1789762737.854374:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762737.854378:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762737.854391:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762737.854421:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504244736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762737.854434:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762737.854438:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762737.854440:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762737.854446:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2664 00000100:00080000:3.0:1789762737.854451:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762737.854455:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762737.854481:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504244608:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789762737.854483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762737.854503:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504244864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762737.854515:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762737.854520:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762737.854522:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762737.854526:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2664 00000100:00080000:3.0:1789762737.854530:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762737.854535:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762737.854540:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762737.854554:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504244992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762737.854565:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762737.854569:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762737.854571:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762737.854574:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2664 00000100:00080000:3.0:1789762737.854579:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762737.854585:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762737.854591:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762737.854603:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762737.854622:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504245120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762737.854640:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762737.854645:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762737/real 1789762737] req@ffff9018d070d880 x1876699504244608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762737.854648:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762737.854656:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762737.854672:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2664 00000100:00080000:3.0:1789762737.854680:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762737.854681:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504244608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762737.854694:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789762737.854721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789762737.854722:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789762737.854732:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762737.854738:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504244608:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762737.854745:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504245248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762737.854751:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504244736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762737.854788:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762737.854828:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762737/real 1789762737] req@ffff9018d070d180 x1876699504244736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762737.854841:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504244736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762737.854851:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762737.854855:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762737.854858:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504244736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762737.854906:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504244864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762737.854929:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762737.854941:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762737/real 1789762737] req@ffff9018d070e680 x1876699504244864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762737.854955:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504244864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762737.854965:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762737.854969:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762737.854973:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504244864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762737.854980:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504244992:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762737.854999:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762737.855009:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762737/real 1789762737] req@ffff9018d070df80 x1876699504244992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762737.855023:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504244992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762737.855103:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762737.855106:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762737.855110:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504244992:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762737.855116:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504245120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762737.855138:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762737.855149:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762737/real 1789762737] req@ffff9018d070ed80 x1876699504245120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762737.855159:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504245120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762737.855167:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762737.855169:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762737.855172:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504245120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762737.855177:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504245248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762737.855190:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762737.855198:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762737/real 1789762737] req@ffff9018d070f480 x1876699504245248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762737.855208:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504245248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762737.855215:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762737.855217:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762737.855220:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504245248:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762738.494185:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762739.518161:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762740.542224:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762741.566198:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762742.590214:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762742.974143:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762742.974154:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762742.974159:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762742.974171:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2669 00000100:00080000:3.0:1789762742.974183:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762742.974191:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762742.974244:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504245376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762742.974271:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762742.974274:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762742.974276:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762742.974280:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2669 00000100:00080000:3.0:1789762742.974283:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762742.974286:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762742.974296:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762742.974306:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504245504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762742.974315:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762742.974318:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762742.974319:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762742.974322:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2669 00000100:00080000:3.0:1789762742.974324:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762742.974329:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504245376:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762742.974330:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762742.974338:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762742.974346:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504245632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762742.974354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762742.974360:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762742.974362:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762742.974367:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2669 00000100:00080000:3.0:1789762742.974372:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762742.974377:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762742.974384:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762742.974401:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504245760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762742.974413:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762742.974417:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762742.974423:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762742.974425:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762742.974432:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2669 00000100:00080000:3.0:1789762742.974438:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762742.974444:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762742.974451:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762742.974461:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762742/real 1789762742] req@ffff9018d070fb80 x1876699504245376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762742.974470:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504245888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762742.974482:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762742.974483:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504245376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762742.974488:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762742.974492:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762742.974502:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2669 00000100:00080000:0.0:1789762742.974504:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762742.974506:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762742.974513:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762742.974514:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762742.974524:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762742.974525:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504245376:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762742.974540:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504246016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762742.974553:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504245504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762742.974585:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762742.974597:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762742/real 1789762742] req@ffff9018d070ca80 x1876699504245504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762742.974607:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504245504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762742.974613:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762742.974616:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762742.974618:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504245504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762742.974622:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504245632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762742.974635:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762742.974641:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762742/real 1789762742] req@ffff9018d070ce00 x1876699504245632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762742.974655:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504245632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762742.974661:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762742.974663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762742.974665:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504245632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762742.974677:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504245760:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762742.974692:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762742.974699:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762742/real 1789762742] req@ffff9018d070e300 x1876699504245760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762742.974708:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504245760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762742.974713:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762742.974715:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762742.974717:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504245760:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762742.974721:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504245888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762742.974733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762742.974740:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762742/real 1789762742] req@ffff9018d070c000 x1876699504245888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762742.974748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504245888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762742.974760:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762742.974762:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762742.974765:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504245888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762742.974778:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504246016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762742.974790:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762742.974796:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762742/real 1789762742] req@ffff9018d070dc00 x1876699504246016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762742.974804:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504246016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762742.974811:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762742.974813:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762742.974815:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504246016:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762743.614164:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762744.638245:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762745.662183:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762746.686214:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762747.710154:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762748.094127:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762748.094138:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762748.094142:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762748.094153:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2674 00000100:00080000:3.0:1789762748.094165:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762748.094173:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762748.094216:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504246144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762748.094244:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762748.094248:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762748.094249:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762748.094253:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2674 00000100:00080000:3.0:1789762748.094256:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762748.094259:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762748.094268:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762748.094281:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504246272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762748.094290:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504246144:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762748.094292:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762748.094300:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762748.094302:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762748.094309:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2674 00000100:00080000:3.0:1789762748.094313:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762748.094317:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762748.094324:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762748.094337:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504246400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762748.094346:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762748.094349:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762748.094351:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762748.094354:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2674 00000100:00080000:3.0:1789762748.094357:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762748.094361:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762748.094366:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762748.094379:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504246528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762748.094409:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762748.094457:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762748/real 1789762748] req@ffff9018d070d500 x1876699504246144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762748.094482:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504246144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762748.094495:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762748.094499:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762748.094503:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504246144:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762748.094529:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504246272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762748.094559:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762748.094577:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762748/real 1789762748] req@ffff9018d070f800 x1876699504246272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762748.094591:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504246272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762748.094616:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762748.094618:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762748.094622:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762748.094625:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789762748.094629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504246272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762748.094638:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504246400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762748.094661:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762748.094664:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762748.094672:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2674 00000100:00080000:3.0:1789762748.094677:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762748.094683:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762748.094691:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762748/real 1789762748] req@ffff9018d070f100 x1876699504246400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789762748.094691:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762748.094714:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504246656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762748.094724:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504246400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762748.094724:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762748.094733:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762748.094737:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762748.094745:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2674 00000100:00080000:3.0:1789762748.094752:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762748.094757:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789762748.094758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789762748.094761:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762748.094764:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762748.094769:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504246400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762748.094773:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504246784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762748.094790:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504246528:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762748.094811:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762748.094819:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762748/real 1789762748] req@ffff9018d070dc00 x1876699504246528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762748.094827:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504246528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762748.094834:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762748.094836:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762748.094838:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504246528:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762748.094841:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504246656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762748.094851:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762748.094857:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762748/real 1789762748] req@ffff9018d070c000 x1876699504246656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762748.094865:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504246656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762748.094873:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762748.094877:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762748.094881:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504246656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762748.094887:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504246784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762748.094907:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762748.094919:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762748/real 1789762748] req@ffff9018d070e300 x1876699504246784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762748.094930:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504246784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762748.094935:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762748.094937:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762748.094939:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504246784:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762748.734208:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762749.758201:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762750.782217:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762751.806218:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762752.830135:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762753.214100:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762753.214108:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762753.214112:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762753.214120:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2679 00000100:00080000:3.0:1789762753.214129:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762753.214135:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762753.214166:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504246912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762753.214188:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762753.214191:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762753.214193:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762753.214195:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2679 00000100:00080000:3.0:1789762753.214198:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762753.214201:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762753.214215:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762753.214225:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504247040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762753.214238:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504246912:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762753.214238:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762753.214248:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762753.214253:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762753.214259:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2679 00000100:00080000:3.0:1789762753.214261:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762753.214264:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762753.214266:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762753.214276:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504247168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762753.214282:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762753.214285:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762753.214286:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762753.214289:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2679 00000100:00080000:3.0:1789762753.214291:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762753.214294:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762753.214298:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762753.214306:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504247296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762753.214312:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762753.214314:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762753.214314:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762753.214317:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2679 00000100:00080000:3.0:1789762753.214327:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762753.214329:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762753.214333:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762753.214339:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762753.214346:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504247424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762753.214355:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762753.214358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762753.214360:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762753.214362:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2679 00000100:00080000:3.0:1789762753.214369:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762753.214371:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762753.214374:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762753.214383:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504247552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762753.214393:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762753/real 1789762753] req@ffff9018d070ce00 x1876699504246912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762753.214414:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504246912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762753.214424:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762753.214427:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762753.214431:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504246912:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762753.214458:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504247040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762753.214482:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762753.214497:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762753/real 1789762753] req@ffff9018d070ca80 x1876699504247040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762753.214512:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504247040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762753.214521:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762753.214524:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762753.214526:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504247040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762753.214532:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504247168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762753.214548:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762753.214556:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762753/real 1789762753] req@ffff9018d070fb80 x1876699504247168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762753.214564:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504247168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762753.214579:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762753.214581:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762753.214584:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504247168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762753.214588:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504247296:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762753.214601:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762753.214608:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762753/real 1789762753] req@ffff9018d070f480 x1876699504247296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762753.214616:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504247296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762753.214622:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762753.214625:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762753.214627:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504247296:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762753.214630:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504247424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762753.214641:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762753.214646:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762753/real 1789762753] req@ffff9018d070ed80 x1876699504247424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762753.214653:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504247424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762753.214658:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762753.214660:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762753.214662:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504247424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762753.214666:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504247552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762753.214676:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762753.214681:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762753/real 1789762753] req@ffff9018d070df80 x1876699504247552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762753.214687:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504247552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762753.214692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762753.214694:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762753.214697:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504247552:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762753.854189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762754.878177:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762755.902162:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762756.926183:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762757.950150:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762758.334097:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762758.334107:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762758.334111:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762758.334120:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2684 00000100:00080000:3.0:1789762758.334130:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762758.334138:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762758.334174:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504247680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762758.334202:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762758.334207:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762758.334209:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762758.334213:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2684 00000100:00080000:3.0:1789762758.334217:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762758.334221:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762758.334231:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762758.334244:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504247808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762758.334263:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762758.334267:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762758.334269:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762758.334273:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2684 00000100:00080000:3.0:1789762758.334277:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762758.334283:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504247680:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762758.334284:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762758.334293:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762758.334308:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504247936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762758.334320:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762758.334324:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762758.334326:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762758.334330:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2684 00000100:00080000:3.0:1789762758.334335:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762758.334339:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762758.334345:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762758.334358:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504248064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762758.334369:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762758.334373:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504247808:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762758.334375:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762758.334381:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762758.334387:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2684 00000100:00080000:3.0:1789762758.334392:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762758.334396:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762758.334403:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762758.334410:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504247936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762758.334417:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504248192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762758.334429:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504248064:192.168.203.158@tcp:8:kworker.0 00000100:00080000:3.0:1789762758.334439:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762758.334444:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762758.334447:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762758.334448:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504248192:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762758.334456:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2684 00000100:00080000:3.0:1789762758.334462:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762758.334467:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762758.334473:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762758.334486:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504248320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762758.334543:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504248320:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762758.974156:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762759.998152:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762761.022146:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762761.406265:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:1.0:1789762761.406301:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762761.406314:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789762761.406343:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789762761.406349:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789762761.406354:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789762761.406360:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789762761.406367:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:2.0:1789762761.406397:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762758/real 1789762761] req@ffff9018d070f480 x1876699504248320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762761.406423:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504248320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762761.406436:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762761.406441:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762761.406446:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504248320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762761.406454:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762758/real 1789762761] req@ffff9018d070ed80 x1876699504248192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762761.406467:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504248192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762761.406475:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762761.406479:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762761.406483:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504248192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762761.406489:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762758/real 1789762761] req@ffff9018d070d880 x1876699504247936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762761.406502:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504247936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762761.406510:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762761.406514:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762761.406517:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504247936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762761.406524:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762758/real 1789762761] req@ffff9018d070df80 x1876699504248064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762761.406536:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504248064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762761.406544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762761.406548:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762761.406551:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504248064:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762761.406557:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762758/real 1789762761] req@ffff9018d070e680 x1876699504247680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762761.406569:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504247680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762761.406577:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762761.406580:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762761.406584:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504247680:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762761.406591:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762758/real 1789762761] req@ffff9018d070d180 x1876699504247808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762761.406603:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504247808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762761.406610:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762761.406613:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762761.406617:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504247808:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762762.046140:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762763.070154:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762763.454133:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762763.454145:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762763.454150:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762763.454163:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2690 00000100:00080000:3.0:1789762763.454177:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762763.454187:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762763.454237:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504248448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762763.454278:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762763.454282:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762763.454284:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762763.454290:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2690 00000100:00080000:3.0:1789762763.454294:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762763.454297:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762763.454308:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762763.454330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504248576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762763.454345:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762763.454348:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789762763.454351:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504248448:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762763.454351:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762763.454361:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2690 00000100:00080000:3.0:1789762763.454366:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762763.454370:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762763.454376:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762763.454392:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504248704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762763.454404:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762763.454408:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762763.454411:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762763.454415:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2690 00000100:00080000:3.0:1789762763.454438:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762763.454443:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789762763.454450:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789762763.454452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762763.454469:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504248832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762763.454482:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762763.454486:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762763.454489:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762763.454494:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2690 00000100:00080000:3.0:1789762763.454499:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762763.454503:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762763.454509:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762763.454510:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762763/real 1789762763] req@ffff9018d070fb80 x1876699504248448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762763.454536:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504248448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762763.454540:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504248960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762763.454560:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762763.454568:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762763.454568:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762763.454577:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789762763.454578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504248448:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762763.454583:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762763.454591:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2690 00000100:00080000:3.0:1789762763.454596:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762763.454600:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762763.454606:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762763.454614:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504248576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762763.454622:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504249088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762763.454646:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762763.454664:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762763/real 1789762763] req@ffff9018d070ca80 x1876699504248576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762763.454683:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504248576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762763.454692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762763.454695:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762763.454699:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504248576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762763.454705:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504248704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762763.454726:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762763.454738:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762763/real 1789762763] req@ffff9018d070ce00 x1876699504248704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762763.454749:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504248704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762763.454757:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762763.454760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762763.454764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504248704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762763.454770:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504248832:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762763.454789:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762763.454801:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762763/real 1789762763] req@ffff9018d070e300 x1876699504248832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762763.454815:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504248832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762763.454830:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762763.454833:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762763.454853:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504248832:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762763.454860:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504248960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762763.454884:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762763.454895:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762763/real 1789762763] req@ffff9018d070c000 x1876699504248960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762763.454913:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504248960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762763.454922:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762763.454926:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762763.454930:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504248960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762763.454952:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504249088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762763.454977:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762763.454990:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762763/real 1789762763] req@ffff9018d070dc00 x1876699504249088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762763.455002:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504249088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762763.455010:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762763.455013:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762763.455017:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504249088:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762764.094340:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762765.118190:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762766.142232:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762767.166302:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762768.190315:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762768.574197:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762768.574212:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762768.574218:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762768.574233:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2695 00000100:00080000:3.0:1789762768.574251:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762768.574261:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762768.574326:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504249216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762768.574368:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762768.574373:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762768.574376:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762768.574383:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2695 00000100:00080000:3.0:1789762768.574387:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762768.574392:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762768.574407:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762768.574424:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504249344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762768.574438:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762768.574443:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762768.574445:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762768.574450:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2695 00000100:00080000:3.0:1789762768.574454:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762768.574462:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504249216:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762768.574465:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762768.574479:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762768.574493:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504249472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762768.574505:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762768.574509:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762768.574511:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762768.574517:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2695 00000100:00080000:3.0:1789762768.574521:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762768.574526:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762768.574532:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762768.574547:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504249600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762768.574561:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762768.574566:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762768.574569:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762768.574574:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2695 00000100:00080000:3.0:1789762768.574578:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762768.574583:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762768.574588:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762768.574595:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762768.574610:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504249728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762768.574624:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762768.574630:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762768.574632:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762768.574637:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2695 00000100:00080000:3.0:1789762768.574642:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762768.574647:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762768.574653:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762768.574680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762768/real 1789762768] req@ffff9018d070f100 x1876699504249216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762768.574687:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504249856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762768.574728:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504249216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762768.574749:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762768.574754:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762768.574760:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504249216:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762768.574771:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504249344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762768.574821:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762768.574837:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762768/real 1789762768] req@ffff9018d070f800 x1876699504249344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762768.574851:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504249344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762768.574860:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762768.574873:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762768.574877:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504249344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762768.574908:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504249472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762768.574930:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762768.574943:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762768/real 1789762768] req@ffff9018d070d500 x1876699504249472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762768.574955:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504249472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762768.574964:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762768.574967:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762768.574970:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504249472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762768.574976:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504249600:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762768.574993:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762768.575004:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762768/real 1789762768] req@ffff9018d070dc00 x1876699504249600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762768.575016:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504249600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762768.575025:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762768.575027:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762768.575030:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504249600:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762768.575074:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504249728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762768.575096:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762768.575128:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762768/real 1789762768] req@ffff9018d070c000 x1876699504249728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762768.575142:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504249728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762768.575154:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762768.575157:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762768.575161:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504249728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762768.575168:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504249856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762768.575189:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762768.575199:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762768/real 1789762768] req@ffff9018d070e300 x1876699504249856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762768.575213:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504249856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762768.575223:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762768.575244:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762768.575247:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504249856:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762769.214219:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762770.238204:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762771.262228:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762772.286245:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762773.310203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762773.694145:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762773.694155:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762773.694160:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762773.694172:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2700 00000100:00080000:0.0:1789762773.694184:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762773.694192:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762773.694234:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504249984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762773.694259:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762773.694262:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762773.694264:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762773.694267:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2700 00000100:00080000:0.0:1789762773.694269:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762773.694272:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762773.694281:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762773.694289:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504250112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762773.694296:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762773.694299:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762773.694300:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762773.694303:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2700 00000100:00080000:0.0:1789762773.694305:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762773.694308:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762773.694311:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762773.694321:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504250240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762773.694332:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504249984:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762773.694349:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762773.694354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762773.694356:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762773.694362:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2700 00000100:00080000:0.0:1789762773.694366:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762773.694370:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762773.694375:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762773.694388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504250368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762773.694399:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762773.694404:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762773.694406:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762773.694411:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762773.694412:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2700 00000100:00080000:0.0:1789762773.694420:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762773.694426:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762773.694432:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762773.694446:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504250496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762773.694457:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762773.694461:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762773.694463:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762773.694467:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762773/real 1789762773] req@ffff9018c8b9bb80 x1876699504249984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762773.694470:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2700 00000100:00080000:0.0:1789762773.694478:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762773.694485:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762773.694491:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762773.694493:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504249984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762773.694504:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789762773.694505:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504250624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762773.694510:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762773.694519:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504249984:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762773.694526:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504250112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762773.694548:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762773.694561:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762773/real 1789762773] req@ffff9018c8b98700 x1876699504250112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762773.694574:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504250112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762773.694583:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762773.694587:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762773.694591:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504250112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762773.694619:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504250240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762773.694635:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762773.694643:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762773/real 1789762773] req@ffff9018c8b99f80 x1876699504250240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762773.694651:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504250240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762773.694657:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762773.694659:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762773.694661:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504250240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762773.694664:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504250368:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762773.694675:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762773.694683:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762773/real 1789762773] req@ffff9018c8b98a80 x1876699504250368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762773.694695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504250368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762773.694704:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762773.694708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762773.694711:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504250368:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762773.694715:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504250496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762773.694730:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762773.694737:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762773/real 1789762773] req@ffff9018c8b99880 x1876699504250496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762773.694746:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504250496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762773.694751:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762773.694753:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762773.694756:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504250496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762773.694775:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504250624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762773.694796:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762773.694808:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762773/real 1789762773] req@ffff9018c8b9b800 x1876699504250624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762773.694821:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504250624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762773.694827:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762773.694830:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762773.694832:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504250624:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762774.334207:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762775.358178:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762776.382219:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762777.406209:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762778.430177:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762778.814156:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762778.814168:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762778.814172:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762778.814184:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2705 00000100:00080000:1.0:1789762778.814195:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762778.814202:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762778.814245:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504250752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762778.814266:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762778.814269:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762778.814270:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762778.814273:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2705 00000100:00080000:1.0:1789762778.814276:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762778.814278:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762778.814286:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762778.814295:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504250880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762778.814302:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762778.814305:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762778.814307:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762778.814311:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2705 00000100:00080000:1.0:1789762778.814314:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762778.814317:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762778.814319:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762778.814326:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504251008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762778.814335:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762778.814338:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762778.814339:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762778.814341:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2705 00000100:00080000:1.0:1789762778.814344:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762778.814349:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762778.814355:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762778.814356:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504250752:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762778.814366:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504251136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762778.814374:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762778.814376:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762778.814377:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762778.814381:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2705 00000100:00080000:1.0:1789762778.814383:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762778.814385:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762778.814389:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762778.814397:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504251264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762778.814404:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762778.814406:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762778.814407:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762778.814410:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2705 00000100:00080000:1.0:1789762778.814414:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762778.814417:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762778.814422:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762778.814428:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504251392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762778.814446:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762778.814494:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762778/real 1789762778] req@ffff9018c66a5880 x1876699504250752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762778.814510:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504250752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762778.814520:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762778.814524:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762778.814527:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504250752:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762778.814533:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504250880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762778.814549:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762778.814557:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762778/real 1789762778] req@ffff9018c66a7100 x1876699504250880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762778.814565:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504250880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762778.814571:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762778.814573:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762778.814575:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504250880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762778.814601:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504251008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762778.814613:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762778.814619:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762778/real 1789762778] req@ffff9018c66a4a80 x1876699504251008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762778.814627:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504251008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762778.814632:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762778.814634:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762778.814636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504251008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762778.814640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504251136:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762778.814653:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762778.814664:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762778/real 1789762778] req@ffff9018c66a7480 x1876699504251136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762778.814676:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504251136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762778.814685:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762778.814689:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762778.814693:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504251136:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762778.814700:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504251264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762778.814736:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762778.814746:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762778/real 1789762778] req@ffff9018c66a6a00 x1876699504251264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762778.814754:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504251264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762778.814760:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762778.814762:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762778.814764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504251264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762778.814768:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504251392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762778.814783:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762778.814789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762778/real 1789762778] req@ffff9018c66a4380 x1876699504251392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762778.814797:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504251392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762778.814801:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762778.814803:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762778.814805:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504251392:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762779.454217:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762780.478191:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762781.502265:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762782.526296:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762783.550217:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762783.934148:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762783.934161:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762783.934167:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762783.934180:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2710 00000100:00080000:2.0:1789762783.934194:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762783.934202:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762783.934248:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504251520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762783.934295:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762783.934300:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762783.934302:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762783.934308:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2710 00000100:00080000:2.0:1789762783.934313:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762783.934319:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762783.934332:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762783.934349:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504251648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762783.934361:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762783.934364:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762783.934366:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762783.934370:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2710 00000100:00080000:2.0:1789762783.934373:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762783.934379:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762783.934384:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762783.934390:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504251520:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762783.934398:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504251776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762783.934407:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762783.934411:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762783.934412:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762783.934416:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2710 00000100:00080000:2.0:1789762783.934422:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762783.934425:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762783.934432:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762783.934441:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504251904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762783.934450:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762783.934453:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762783.934454:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762783.934457:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2710 00000100:00080000:2.0:1789762783.934459:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762783.934463:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762783.934467:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762783.934479:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504252032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762783.934486:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762783.934489:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762783.934492:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762783.934495:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2710 00000100:00080000:2.0:1789762783.934499:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762783.934502:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762783.934507:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762783.934515:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504252160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762783.934539:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762783.934579:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762783/real 1789762783] req@ffff9018d0f24700 x1876699504251520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762783.934596:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504251520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762783.934758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762783.934764:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762783.934770:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504251520:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762783.934780:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504251648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762783.934872:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762783.934888:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762783/real 1789762783] req@ffff9018d0f26680 x1876699504251648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762783.934900:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504251648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762783.934910:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762783.934913:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762783.934917:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504251648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762783.934947:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504251776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762783.934969:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762783.934981:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762783/real 1789762783] req@ffff9018d0f27480 x1876699504251776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762783.934995:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504251776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762783.935001:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762783.935003:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762783.935005:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504251776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762783.935010:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504251904:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762783.935026:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762783.935067:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762783/real 1789762783] req@ffff9018d0f25500 x1876699504251904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762783.935078:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504251904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762783.935085:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762783.935088:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762783.935092:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504251904:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762783.935097:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504252032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762783.935114:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762783.935122:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762783/real 1789762783] req@ffff9018d0f24000 x1876699504252032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762783.935131:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504252032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762783.935137:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762783.935138:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762783.935141:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504252032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762783.935144:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504252160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762783.935154:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762783.935159:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762783/real 1789762783] req@ffff9018d0f24e00 x1876699504252160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762783.935167:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504252160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762783.935172:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762783.935182:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762783.935184:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504252160:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762784.574176:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762785.598178:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762786.622188:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762787.646243:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762788.670169:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762789.054199:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762789.054210:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762789.054215:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762789.054227:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2715 00000100:00080000:1.0:1789762789.054238:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762789.054245:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762789.054285:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504252288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762789.054309:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762789.054312:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762789.054313:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762789.054317:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2715 00000100:00080000:1.0:1789762789.054320:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762789.054324:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762789.054333:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762789.054347:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504252416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762789.054354:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504252288:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762789.054366:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762789.054371:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762789.054373:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762789.054378:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2715 00000100:00080000:1.0:1789762789.054382:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762789.054387:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762789.054393:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762789.054406:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504252544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762789.054416:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762789.054420:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762789.054422:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762789.054426:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2715 00000100:00080000:1.0:1789762789.054438:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789762789.054444:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762789.054448:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762789.054456:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762789.054469:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504252672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762789.054487:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762789.054491:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762789.054493:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762789.054497:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2715 00000100:00100000:3.0:1789762789.054503:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762789/real 1789762789] req@ffff9018c66a4e00 x1876699504252288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762789.054504:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762789.054513:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762789.054523:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762789.054550:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504252288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762789.054554:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504252800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762789.054578:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762789.054579:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762789.054587:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762789.054590:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789762789.054591:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762789.054599:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504252288:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762789.054600:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2715 00000100:00080000:1.0:1789762789.054609:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762789.054615:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762789.054621:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762789.054633:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504252928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762789.054639:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504252416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762789.054688:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762789.054703:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762789/real 1789762789] req@ffff9018c66a5500 x1876699504252416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762789.054717:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504252416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762789.054726:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762789.054729:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762789.054733:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504252416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762789.054739:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504252544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762789.054759:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762789.054772:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762789/real 1789762789] req@ffff9018c66a6680 x1876699504252544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762789.054785:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504252544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762789.054799:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762789.054803:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762789.054807:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504252544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762789.054814:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504252672:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762789.054835:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762789.054847:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762789/real 1789762789] req@ffff9018c66a4700 x1876699504252672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762789.054858:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504252672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762789.054867:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762789.054870:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762789.054874:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504252672:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762789.054880:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504252800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762789.054897:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762789.054908:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762789/real 1789762789] req@ffff9018c66a6300 x1876699504252800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762789.054920:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504252800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762789.054929:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762789.054932:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762789.054936:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504252800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762789.054954:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504252928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762789.054973:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762789.054985:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762789/real 1789762789] req@ffff9018c66a5180 x1876699504252928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762789.054998:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504252928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762789.055006:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762789.055010:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762789.055014:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504252928:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762789.694229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762790.718205:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762791.742242:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762792.766260:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762793.790235:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762794.174156:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762794.174167:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762794.174171:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762794.174184:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2720 00000100:00080000:0.0:1789762794.174196:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762794.174204:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762794.174249:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504253056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762794.174277:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762794.174281:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762794.174282:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762794.174286:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2720 00000100:00080000:0.0:1789762794.174288:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762794.174290:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762794.174300:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762794.174312:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504253184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762794.174323:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762794.174326:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762794.174327:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762794.174330:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2720 00000100:00080000:0.0:1789762794.174332:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762794.174335:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762794.174337:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762794.174344:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504253312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762794.174350:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762794.174353:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762794.174355:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762794.174358:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2720 00000100:00080000:0.0:1789762794.174360:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762794.174362:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762794.174366:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762794.174376:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504253440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762794.174376:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504253056:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762794.174391:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762794.174393:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762794.174395:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762794.174397:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2720 00000100:00080000:0.0:1789762794.174399:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762794.174401:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762794.174404:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762794.174412:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504253568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762794.174418:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762794.174420:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762794.174421:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762794.174424:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2720 00000100:00080000:0.0:1789762794.174426:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762794.174429:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762794.174432:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762794.174438:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504253696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762794.174457:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762794.174502:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762794/real 1789762794] req@ffff9018c8b9b100 x1876699504253056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762794.174518:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504253056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762794.174538:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762794.174541:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762794.174544:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504253056:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762794.174549:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504253184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762794.174564:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762794.174573:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762794/real 1789762794] req@ffff9018c8b9b480 x1876699504253184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762794.174581:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504253184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762794.174587:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762794.174588:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762794.174590:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504253184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762794.174605:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504253312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762794.174634:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762794.174641:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762794/real 1789762794] req@ffff9018c8b98000 x1876699504253312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762794.174649:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504253312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762794.174654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762794.174657:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762794.174669:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504253312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762794.174692:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504253440:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762794.174703:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762794.174710:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762794/real 1789762794] req@ffff9018c8b9aa00 x1876699504253440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762794.174722:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504253440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762794.174727:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762794.174729:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762794.174731:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504253440:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762794.174734:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504253568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762794.174744:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762794.174751:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762794/real 1789762794] req@ffff9018c8b98e00 x1876699504253568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762794.174758:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504253568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762794.174763:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762794.174765:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762794.174767:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504253568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762794.174770:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504253696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762794.174780:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762794.174785:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762794/real 1789762794] req@ffff9018c8b9a680 x1876699504253696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762794.174793:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504253696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762794.174803:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762794.174805:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762794.174807:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504253696:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762794.814214:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762795.838206:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762796.862175:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762797.886175:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762798.910168:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762799.294242:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762799.294280:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762799.294286:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762799.294300:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2725 00000100:00080000:1.0:1789762799.294314:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762799.294325:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762799.294376:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504253824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762799.294424:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762799.294428:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762799.294430:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762799.294435:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2725 00000100:00080000:1.0:1789762799.294440:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762799.294445:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762799.294474:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762799.294499:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504253824:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762799.294500:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504253952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762799.294537:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762799.294541:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762799.294543:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762799.294548:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2725 00000100:00080000:1.0:1789762799.294552:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762799.294557:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762799.294563:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762799.294588:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504254080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762799.294597:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762799.294600:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762799.294602:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762799.294605:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2725 00000800:00000100:3.0:1789762799.294611:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762799.294612:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762799.294618:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762799.294623:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762799.294634:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504254208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762799.294643:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762799.294647:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762799.294649:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762799.294653:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2725 00000100:00080000:1.0:1789762799.294658:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762799.294663:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762799.294669:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762799.294669:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762799/real 1789762799] req@ffff9018c66a7800 x1876699504253824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762799.294687:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504254336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762799.294693:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504253824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762799.294715:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762799.294717:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762799.294722:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762799.294725:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789762799.294727:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762799.294730:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504253824:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762799.294736:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2725 00000100:00080000:1.0:1789762799.294741:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762799.294745:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762799.294751:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762799.294759:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504253952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762799.294765:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504254464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762799.294783:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762799.294795:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762799/real 1789762799] req@ffff9018c66a7b80 x1876699504253952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762799.294805:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504253952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762799.294812:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762799.294815:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762799.294818:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504253952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762799.294822:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504254080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762799.294834:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762799.294840:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762799/real 1789762799] req@ffff9018c66a4000 x1876699504254080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762799.294848:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504254080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762799.294853:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762799.294855:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762799.294857:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504254080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762799.294861:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504254208:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762799.294871:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762799.294877:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762799/real 1789762799] req@ffff9018c66a5f80 x1876699504254208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762799.294885:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504254208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762799.294890:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762799.294892:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762799.294894:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504254208:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762799.294897:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504254336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762799.294906:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762799.294913:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762799/real 1789762799] req@ffff9018c66a6d80 x1876699504254336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762799.294921:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504254336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762799.294926:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762799.294928:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762799.294930:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504254336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762799.294943:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504254464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762799.294953:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762799.294959:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762799/real 1789762799] req@ffff9018c66a5c00 x1876699504254464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762799.294966:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504254464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762799.294971:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762799.294972:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762799.294974:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504254464:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762799.934236:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762800.958216:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762801.982250:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762803.006217:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762804.030166:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762804.414092:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762804.414100:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762804.414103:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762804.414112:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2730 00000100:00080000:1.0:1789762804.414120:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762804.414126:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762804.414166:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504254592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762804.414185:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762804.414188:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762804.414189:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762804.414192:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2730 00000100:00080000:1.0:1789762804.414194:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762804.414197:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762804.414204:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762804.414213:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504254720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762804.414220:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762804.414222:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762804.414223:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762804.414225:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2730 00000100:00080000:1.0:1789762804.414228:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762804.414231:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762804.414234:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762804.414241:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504254848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762804.414247:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762804.414250:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762804.414251:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762804.414256:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2730 00000100:00100000:3.0:1789762804.414258:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504254592:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762804.414259:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762804.414265:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762804.414269:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762804.414277:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504254976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762804.414283:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762804.414286:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762804.414287:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762804.414290:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2730 00000100:00080000:1.0:1789762804.414292:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762804.414295:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762804.414298:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762804.414305:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504255104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762804.414315:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762804.414319:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1789762804.414320:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762804.414322:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762804.414326:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2730 00000100:00080000:1.0:1789762804.414330:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762804.414333:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762804.414336:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762804.414345:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504255232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762804.414360:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762804/real 1789762804] req@ffff9018c66a5c00 x1876699504254592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762804.414379:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504254592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762804.414387:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762804.414390:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762804.414393:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504254592:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762804.414397:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504254720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762804.414411:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762804.414418:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762804/real 1789762804] req@ffff9018c66a6d80 x1876699504254720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762804.414426:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504254720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762804.414432:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762804.414434:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762804.414436:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504254720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762804.414454:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504254848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762804.414464:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762804.414470:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762804/real 1789762804] req@ffff9018c66a5f80 x1876699504254848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762804.414477:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504254848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762804.414496:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762804.414498:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762804.414501:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504254848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762804.414504:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504254976:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762804.414516:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762804.414523:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762804/real 1789762804] req@ffff9018c66a4000 x1876699504254976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762804.414547:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504254976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762804.414554:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762804.414556:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762804.414558:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504254976:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762804.414570:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504255104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762804.414590:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762804.414603:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762804/real 1789762804] req@ffff9018c66a7b80 x1876699504255104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762804.414616:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504255104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762804.414625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762804.414628:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762804.414631:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504255104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762804.414635:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504255232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762804.414650:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762804.414658:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762804/real 1789762804] req@ffff9018c66a7800 x1876699504255232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762804.414665:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504255232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762804.414670:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762804.414672:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762804.414674:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504255232:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762805.054221:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762806.078215:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762807.102204:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762808.126179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762809.150227:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762809.534169:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762809.534180:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762809.534184:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762809.534196:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2736 00000100:00080000:0.0:1789762809.534208:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762809.534216:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762809.534261:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504255360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762809.534288:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762809.534291:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762809.534293:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762809.534296:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2736 00000100:00080000:0.0:1789762809.534298:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762809.534301:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762809.534311:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762809.534322:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504255488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762809.534334:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762809.534337:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762809.534338:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762809.534343:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2736 00000100:00080000:0.0:1789762809.534350:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789762809.534352:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504255360:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762809.534355:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762809.534365:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762809.534375:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504255616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762809.534381:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762809.534385:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762809.534387:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762809.534390:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2736 00000100:00080000:0.0:1789762809.534393:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762809.534396:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762809.534400:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762809.534412:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504255744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762809.534419:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762809.534423:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1789762809.534431:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762809.534432:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762809.534445:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2736 00000100:00080000:0.0:1789762809.534451:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762809.534454:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762809.534460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762809.534469:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504255872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762809.534478:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762809.534483:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789762809.534484:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762809/real 1789762809] req@ffff9018c8b9a300 x1876699504255360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762809.534486:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762809.534509:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2736 00000100:00080000:0.0:1789762809.534515:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762809.534526:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762809.534537:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762809.534548:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504255360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762809.534549:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504256000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762809.534632:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762809.534637:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762809.534641:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504255360:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762809.534674:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504255488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762809.534695:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762809.534705:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762809/real 1789762809] req@ffff9018c8b9ad80 x1876699504255488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762809.534714:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504255488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762809.534721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762809.534723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762809.534725:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504255488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762809.534730:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504255616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762809.534742:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762809.534748:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762809/real 1789762809] req@ffff9018c8b99500 x1876699504255616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762809.534756:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504255616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762809.534761:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762809.534763:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762809.534765:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504255616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762809.534769:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504255744:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762809.534780:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762809.534786:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762809/real 1789762809] req@ffff9018c8b99c00 x1876699504255744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762809.534798:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504255744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762809.534803:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762809.534805:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762809.534807:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504255744:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762809.534811:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504255872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762809.534821:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762809.534827:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762809/real 1789762809] req@ffff9018c8b98380 x1876699504255872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762809.534834:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504255872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762809.534839:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762809.534841:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762809.534843:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504255872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762809.534846:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504256000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762809.534856:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762809.534862:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762809/real 1789762809] req@ffff9018c8b9a680 x1876699504256000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762809.534868:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504256000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762809.534877:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762809.534879:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762809.534881:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504256000:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762810.174193:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762811.198195:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762812.222175:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762813.246217:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762814.270244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762814.654159:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762814.654167:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762814.654170:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762814.654180:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2741 00000100:00080000:1.0:1789762814.654189:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762814.654194:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762814.654225:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504256128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762814.654245:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762814.654248:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762814.654250:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762814.654253:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2741 00000100:00080000:1.0:1789762814.654255:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762814.654257:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762814.654265:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762814.654273:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504256256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762814.654279:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762814.654282:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762814.654283:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762814.654285:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2741 00000100:00080000:1.0:1789762814.654287:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762814.654288:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762814.654291:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762814.654297:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504256384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762814.654305:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762814.654308:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762814.654309:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762814.654312:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2741 00000100:00080000:1.0:1789762814.654314:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762814.654317:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762814.654320:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762814.654329:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504256512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762814.654339:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504256128:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762814.654345:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762814.654352:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762814.654355:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762814.654361:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2741 00000100:00080000:1.0:1789762814.654365:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762814.654372:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762814.654379:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762814.654398:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504256640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762814.654407:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762814.654411:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762814.654414:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762814.654419:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2741 00000100:00080000:1.0:1789762814.654423:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762814.654426:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762814.654430:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789762814.654433:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762814.654466:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504256768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762814.654508:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762814/real 1789762814] req@ffff9018c66a5180 x1876699504256128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762814.654528:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504256128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762814.654538:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762814.654542:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762814.654545:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504256128:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762814.654551:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504256256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762814.654570:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762814.654579:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762814/real 1789762814] req@ffff9018c66a6300 x1876699504256256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762814.654587:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504256256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762814.654593:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762814.654595:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762814.654598:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504256256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762814.654602:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504256384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762814.654613:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762814.654619:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762814/real 1789762814] req@ffff9018c66a4700 x1876699504256384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762814.654626:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504256384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762814.654631:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762814.654633:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762814.654635:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504256384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762814.654662:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504256512:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762814.654674:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762814.654681:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762814/real 1789762814] req@ffff9018c66a6680 x1876699504256512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762814.654687:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504256512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762814.654693:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762814.654695:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762814.654697:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504256512:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762814.654701:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504256640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762814.654711:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762814.654717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762814/real 1789762814] req@ffff9018c66a5500 x1876699504256640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762814.654725:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504256640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762814.654730:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762814.654732:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762814.654734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504256640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762814.654738:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504256768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762814.654748:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762814.654753:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762814/real 1789762814] req@ffff9018c66a4e00 x1876699504256768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762814.654760:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504256768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762814.654765:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762814.654766:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762814.654768:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504256768:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762815.294260:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762816.318271:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762817.342158:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762818.366150:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762819.390159:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762819.774101:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762819.774110:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762819.774113:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762819.774121:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2746 00000100:00080000:1.0:1789762819.774130:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762819.774135:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762819.774168:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504256896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762819.774194:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762819.774198:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762819.774200:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762819.774204:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2746 00000100:00080000:1.0:1789762819.774208:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762819.774212:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762819.774222:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762819.774235:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504257024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762819.774237:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504256896:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762819.774245:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762819.774248:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762819.774250:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762819.774253:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2746 00000100:00080000:1.0:1789762819.774256:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762819.774259:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762819.774262:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762819.774270:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504257152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762819.774278:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762819.774282:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762819.774283:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762819.774287:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2746 00000100:00080000:1.0:1789762819.774289:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762819.774293:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789762819.774295:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:1.0:1789762819.774297:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762819.774305:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504257280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762819.774314:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762819.774317:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762819.774319:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762819.774322:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2746 00000100:00080000:1.0:1789762819.774325:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762819.774328:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762819.774331:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762819.774333:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762819/real 1789762819] req@ffff9018c66a4380 x1876699504256896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762819.774340:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504257408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762819.774348:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789762819.774351:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504256896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762819.774352:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762819.774354:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762819.774357:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2746 00000100:00080000:1.0:1789762819.774360:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762819.774363:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789762819.774363:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1789762819.774367:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789762819.774367:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762819.774371:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504256896:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762819.774375:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504257536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762819.774393:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504257024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762819.774432:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762819.774447:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762819/real 1789762819] req@ffff9018c66a6a00 x1876699504257024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762819.774461:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504257024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762819.774470:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762819.774474:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762819.774477:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504257024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762819.774483:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504257152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762819.774506:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762819.774519:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762819/real 1789762819] req@ffff9018c66a7480 x1876699504257152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762819.774529:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504257152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762819.774537:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762819.774540:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762819.774543:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504257152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762819.774551:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504257280:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789762819.774575:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762819.774589:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762819/real 1789762819] req@ffff9018c66a4a80 x1876699504257280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762819.774602:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504257280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762819.774611:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762819.774614:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762819.774618:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504257280:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789762819.774623:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504257408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762819.774643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762819.774735:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762819/real 1789762819] req@ffff9018c66a7100 x1876699504257408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762819.774749:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504257408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762819.774759:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762819.774762:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762819.774766:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504257408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762819.774874:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504257536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762819.774944:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762819.774961:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762819/real 1789762819] req@ffff9018c66a5880 x1876699504257536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762819.774975:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504257536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762819.774991:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762819.774994:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762819.774998:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504257536:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762820.414174:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762821.438216:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762822.462198:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762823.486212:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762824.510224:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762824.894170:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762824.894183:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762824.894187:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762824.894201:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2751 00000100:00080000:3.0:1789762824.894215:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762824.894224:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762824.894272:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504257664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762824.894306:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762824.894310:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762824.894312:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762824.894317:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2751 00000100:00080000:3.0:1789762824.894323:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762824.894328:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762824.894340:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762824.894344:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504257664:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762824.894356:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504257792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762824.894387:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762824.894393:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762824.894396:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762824.894402:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2751 00000100:00080000:3.0:1789762824.894423:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762824.894427:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762824.894432:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762824.894447:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504257920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762824.894456:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762824.894460:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762824.894462:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762824.894469:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2751 00000100:00080000:3.0:1789762824.894475:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762824.894481:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504257792:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762824.894482:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762824.894493:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762824.894510:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504258048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762824.894517:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504257920:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762824.894529:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762824.894535:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762824.894537:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762824.894544:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2751 00000100:00100000:2.0:1789762824.894545:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504258048:192.168.203.158@tcp:8:kworker.0 00000100:00080000:3.0:1789762824.894551:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762824.894560:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762824.894567:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762824.894581:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504258176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762824.894598:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762824.894602:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762824.894604:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762824.894611:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2751 00000100:00100000:2.0:1789762824.894615:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504258176:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762824.894617:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762824.894626:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762824.894632:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762824.894648:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504258304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762824.894674:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504258304:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762825.534178:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762826.558221:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762827.582206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789762827.966317:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:0.0:1789762827.966382:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789762827.966406:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789762827.966452:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789762827.966463:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789762827.966474:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789762827.966487:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789762827.966509:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:2.0:1789762827.966523:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762824/real 1789762827] req@ffff9018d070e680 x1876699504258176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762827.966556:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504258176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762827.966568:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762827.966573:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762827.966578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504258176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762827.966587:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762824/real 1789762827] req@ffff9018d070d180 x1876699504258048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762827.966601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504258048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762827.966609:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762827.966613:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762827.966617:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504258048:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762827.966625:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762824/real 1789762827] req@ffff9018d070ca80 x1876699504257792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762827.966637:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504257792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762827.966645:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762827.966649:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762827.966652:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504257792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762827.966664:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762824/real 1789762827] req@ffff9018d070fb80 x1876699504257920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762827.966677:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504257920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762827.966685:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762827.966689:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762827.966692:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504257920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762827.966700:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762824/real 1789762827] req@ffff9018d070ce00 x1876699504257664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762827.966729:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504257664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762827.966738:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762827.966741:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762827.966745:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504257664:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762827.966784:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762824/real 1789762827] req@ffff9018d070df80 x1876699504258304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762827.966797:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504258304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762827.966805:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762827.966809:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762827.966812:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504258304:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762828.606243:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762829.630263:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762830.014188:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762830.014199:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762830.014204:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762830.014216:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2756 00000100:00080000:1.0:1789762830.014228:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762830.014235:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762830.014276:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504258432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762830.014313:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762830.014318:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762830.014320:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762830.014326:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2756 00000100:00080000:1.0:1789762830.014330:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762830.014335:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762830.014359:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762830.014377:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504258560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762830.014387:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504258432:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762830.014392:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762830.014397:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762830.014399:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762830.014405:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2756 00000100:00080000:1.0:1789762830.014409:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762830.014414:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762830.014421:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762830.014436:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504258688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762830.014448:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762830.014452:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762830.014454:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762830.014459:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2756 00000100:00080000:1.0:1789762830.014463:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762830.014468:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762830.014474:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789762830.014487:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762830.014489:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504258816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762830.014502:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762830.014506:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762830.014509:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762830.014514:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2756 00000100:00080000:1.0:1789762830.014518:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762830.014523:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762830.014528:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762830.014542:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504258944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762830.014551:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762830/real 1789762830] req@ffff9018c66a5880 x1876699504258432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762830.014556:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762830.014565:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762830.014570:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762830.014577:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2756 00000100:00080000:1.0:1789762830.014582:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762830.014588:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762830.014588:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504258432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762830.014598:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762830.014611:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789762830.014616:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504259072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762830.014621:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762830.014628:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504258432:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762830.014668:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504258560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762830.014702:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762830.014718:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762830/real 1789762830] req@ffff9018c66a7100 x1876699504258560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762830.014733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504258560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762830.014742:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762830.014745:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762830.014748:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504258560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762830.014755:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504258688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762830.014778:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762830.014793:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762830/real 1789762830] req@ffff9018c66a4a80 x1876699504258688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762830.014810:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504258688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762830.014819:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762830.014823:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762830.014827:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504258688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762830.015124:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504258816:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762830.015219:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762830.015254:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762830/real 1789762830] req@ffff9018c66a7480 x1876699504258816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762830.015276:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504258816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762830.015290:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762830.015295:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762830.015300:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504258816:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762830.015310:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504258944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762830.015335:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762830.015366:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762830/real 1789762830] req@ffff9018c66a6a00 x1876699504258944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762830.015379:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504258944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762830.015388:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762830.015392:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762830.015396:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504258944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762830.015403:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504259072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762830.015427:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762830.015439:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762830/real 1789762830] req@ffff9018c66a4380 x1876699504259072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762830.015450:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504259072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762830.015458:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762830.015461:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762830.015464:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504259072:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762830.654257:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762831.678234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762832.702209:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762833.726227:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762834.750180:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762835.134173:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762835.134183:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762835.134187:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762835.134198:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2761 00000100:00080000:0.0:1789762835.134211:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762835.134219:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762835.134265:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504259200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762835.134297:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762835.134312:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762835.134313:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762835.134317:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2761 00000100:00080000:0.0:1789762835.134320:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762835.134325:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762835.134335:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762835.134351:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504259328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762835.134353:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504259200:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762835.134363:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762835.134367:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762835.134369:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762835.134372:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2761 00000100:00080000:0.0:1789762835.134374:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762835.134376:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762835.134380:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762835.134387:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504259456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762835.134393:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762835.134396:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762835.134398:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762835.134400:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2761 00000100:00080000:0.0:1789762835.134404:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762835.134407:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762835.134411:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762835.134418:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504259584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762835.134425:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762835.134427:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762835.134428:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762835.134431:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2761 00000100:00080000:0.0:1789762835.134433:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762835.134437:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762835.134442:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762835.134443:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762835.134451:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504259712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762835.134458:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762835.134461:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762835.134462:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762835.134466:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2761 00000100:00080000:0.0:1789762835.134469:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762835.134472:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762835.134475:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762835.134481:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504259840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762835.134500:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762835/real 1789762835] req@ffff9018c8b98e00 x1876699504259200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762835.134522:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504259200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762835.134541:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762835.134546:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762835.134550:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504259200:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762835.134581:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504259328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762835.134618:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762835.134632:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762835/real 1789762835] req@ffff9018c8b9aa00 x1876699504259328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762835.134646:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504259328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762835.134655:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762835.134659:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762835.134662:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504259328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762835.134669:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504259456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762835.134689:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762835.134700:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762835/real 1789762835] req@ffff9018c8b98000 x1876699504259456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762835.134712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504259456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762835.134721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762835.134724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762835.134728:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504259456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762835.134734:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504259584:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762835.134754:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762835.134765:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762835/real 1789762835] req@ffff9018c8b9b480 x1876699504259584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762835.134778:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504259584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762835.134787:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762835.134790:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762835.134794:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504259584:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762835.134800:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504259712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762835.134819:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762835.134829:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762835/real 1789762835] req@ffff9018c8b9b100 x1876699504259712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762835.134841:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504259712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762835.134850:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762835.134853:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762835.134857:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504259712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762835.134864:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504259840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762835.134883:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762835.134894:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762835/real 1789762835] req@ffff9018c8b9b800 x1876699504259840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762835.134907:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504259840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762835.134916:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762835.134920:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762835.134923:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504259840:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762835.774220:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762836.798225:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762837.822204:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762838.846179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762839.870206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762840.254097:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762840.254109:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762840.254113:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762840.254124:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2766 00000100:00080000:0.0:1789762840.254137:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762840.254144:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762840.254188:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504259968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762840.254215:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762840.254218:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762840.254219:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762840.254223:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2766 00000100:00080000:0.0:1789762840.254225:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762840.254227:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762840.254238:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762840.254248:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504260096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762840.254258:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762840.254271:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762840.254273:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762840.254276:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2766 00000100:00080000:0.0:1789762840.254279:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762840.254283:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762840.254294:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504259968:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789762840.254296:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762840.254312:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504260224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762840.254319:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762840.254322:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762840.254324:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762840.254327:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2766 00000100:00080000:0.0:1789762840.254329:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762840.254332:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762840.254336:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762840.254348:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504260352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762840.254356:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762840.254359:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762840.254360:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762840.254362:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2766 00000100:00080000:0.0:1789762840.254365:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762840.254368:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762840.254371:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762840.254377:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762840.254386:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504260480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762840.254399:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762840.254402:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762840.254404:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762840.254407:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2766 00000100:00080000:0.0:1789762840.254410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762840.254412:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762840.254419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762840.254424:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762840/real 1789762840] req@ffff9018c8b99880 x1876699504259968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762840.254434:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504260608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762840.254443:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504259968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762840.254455:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762840.254460:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762840.254466:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504259968:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762840.254490:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504260096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762840.254515:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762840.254529:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762840/real 1789762840] req@ffff9018c8b98a80 x1876699504260096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762840.254551:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504260096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762840.254557:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762840.254559:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762840.254561:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504260096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762840.254566:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504260224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762840.254601:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762840.254615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762840/real 1789762840] req@ffff9018c8b99f80 x1876699504260224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762840.254627:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504260224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762840.254636:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762840.254639:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762840.254643:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504260224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762840.254649:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504260352:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762840.254669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762840.254680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762840/real 1789762840] req@ffff9018c8b98700 x1876699504260352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762840.254691:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504260352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762840.254704:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762840.254707:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762840.254710:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504260352:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762840.254716:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504260480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762840.254745:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762840.254756:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762840/real 1789762840] req@ffff9018c8b9bb80 x1876699504260480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762840.254767:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504260480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762840.254775:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762840.254779:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762840.254782:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504260480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762840.254789:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504260608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762840.254808:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762840.254820:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762840/real 1789762840] req@ffff9018c8b9b800 x1876699504260608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762840.254832:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504260608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762840.254840:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762840.254843:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762840.254847:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504260608:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762840.894203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762841.918131:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762842.942171:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762843.966245:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762844.990265:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762845.374181:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762845.374189:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762845.374192:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762845.374200:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2771 00000100:00080000:1.0:1789762845.374210:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762845.374216:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762845.374265:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504260736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762845.374289:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762845.374291:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762845.374293:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762845.374296:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2771 00000100:00080000:1.0:1789762845.374298:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762845.374300:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762845.374309:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762845.374318:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504260864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762845.374331:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762845.374335:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504260736:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762845.374336:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762845.374338:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762845.374346:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2771 00000100:00080000:1.0:1789762845.374348:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762845.374350:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762845.374353:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762845.374363:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504260992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762845.374369:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762845.374372:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762845.374374:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762845.374377:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2771 00000100:00080000:1.0:1789762845.374379:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762845.374381:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762845.374385:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762845.374394:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504261120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762845.374400:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762845.374406:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762845.374409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762845.374411:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762845.374415:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2771 00000100:00080000:1.0:1789762845.374419:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762845.374421:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762845.374424:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762845.374431:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504261248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762845.374439:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762845/real 1789762845] req@ffff9018c66a4e00 x1876699504260736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762845.374440:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762845.374446:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762845.374449:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762845.374456:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2771 00000100:00080000:1.0:1789762845.374459:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762845.374462:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762845.374465:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504260736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762845.374467:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762845.374478:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789762845.374480:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504261376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762845.374484:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762845.374493:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504260736:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762845.374517:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504260864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762845.374534:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762845.374542:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762845/real 1789762845] req@ffff9018c66a5500 x1876699504260864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762845.374550:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504260864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762845.374555:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762845.374557:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762845.374559:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504260864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762845.374565:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504260992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762845.374574:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762845.374580:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762845/real 1789762845] req@ffff9018c66a6680 x1876699504260992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762845.374587:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504260992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762845.374592:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762845.374594:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762845.374596:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504260992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762845.374599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504261120:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762845.374609:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762845.374614:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762845/real 1789762845] req@ffff9018c66a4700 x1876699504261120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762845.374621:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504261120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762845.374626:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762845.374627:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762845.374629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504261120:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762845.374632:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504261248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762845.374641:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762845.374646:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762845/real 1789762845] req@ffff9018c66a6300 x1876699504261248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762845.374654:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504261248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762845.374658:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762845.374660:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762845.374662:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504261248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762845.374665:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504261376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762845.374674:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762845.374679:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762845/real 1789762845] req@ffff9018c66a5180 x1876699504261376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762845.374685:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504261376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762845.374689:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762845.374691:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762845.374693:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504261376:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762846.014175:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762847.038166:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762848.062197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762849.086154:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762850.110203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762850.494145:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762850.494156:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762850.494161:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762850.494172:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2777 00000100:00080000:3.0:1789762850.494182:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762850.494204:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762850.494243:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504261504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762850.494270:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762850.494274:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762850.494276:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762850.494280:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2777 00000100:00080000:3.0:1789762850.494284:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762850.494289:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762850.494299:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762850.494313:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504261504:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762850.494315:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504261632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762850.494331:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762850.494334:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762850.494336:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762850.494340:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2777 00000100:00080000:3.0:1789762850.494343:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762850.494348:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762850.494353:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762850.494365:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504261760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762850.494375:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762850.494380:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762850.494383:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762850.494387:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762850.494393:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2777 00000100:00080000:3.0:1789762850.494399:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762850.494403:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762850.494411:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762850.494414:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762850/real 1789762850] req@ffff9018d070d880 x1876699504261504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762850.494429:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504261888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762850.494437:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504261504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762850.494443:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762850.494448:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762850.494449:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762850.494452:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762850.494454:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762850.494457:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504261504:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762850.494463:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2777 00000100:00080000:3.0:1789762850.494468:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762850.494474:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762850.494480:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762850.494484:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504261632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762850.494494:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504262016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762850.494507:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762850.494511:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1789762850.494513:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762850.494513:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762850.494522:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2777 00000100:00080000:3.0:1789762850.494526:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762850.494530:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762850/real 1789762850] req@ffff9018d070ed80 x1876699504261632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762850.494534:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762850.494544:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762850.494549:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504261632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762850.494557:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789762850.494559:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504262144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762850.494562:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762850.494569:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504261632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762850.494580:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504261760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762850.494599:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762850.494607:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762850/real 1789762850] req@ffff9018d070f480 x1876699504261760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762850.494615:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504261760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762850.494625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762850.494627:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762850.494629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504261760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762850.494633:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504261888:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762850.494644:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762850.494649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762850/real 1789762850] req@ffff9018d070df80 x1876699504261888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762850.494656:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504261888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762850.494662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762850.494663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762850.494666:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504261888:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762850.494676:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504262016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762850.494686:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762850.494692:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762850/real 1789762850] req@ffff9018d070ce00 x1876699504262016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762850.494700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504262016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762850.494704:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762850.494706:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762850.494708:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504262016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762850.494712:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504262144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762850.494722:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762850.494727:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762850/real 1789762850] req@ffff9018d070fb80 x1876699504262144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762850.494734:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504262144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762850.494739:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762850.494741:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762850.494743:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504262144:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762851.134171:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762852.158168:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762853.182161:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762854.206180:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762855.230171:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762855.614133:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762855.614141:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762855.614145:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762855.614170:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2782 00000100:00080000:3.0:1789762855.614181:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762855.614194:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762855.614225:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504262272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762855.614250:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762855.614253:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762855.614254:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762855.614257:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2782 00000100:00080000:3.0:1789762855.614260:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762855.614263:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762855.614271:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762855.614281:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504262400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762855.614288:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762855.614291:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762855.614292:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762855.614297:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2782 00000100:00100000:0.0:1789762855.614298:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504262272:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762855.614300:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762855.614308:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762855.614312:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762855.614322:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504262528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762855.614334:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762855.614337:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762855.614338:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762855.614340:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2782 00000100:00080000:3.0:1789762855.614343:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762855.614345:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762855.614349:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789762855.614359:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762855.614360:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504262656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762855.614370:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762855.614373:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762855.614375:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762855.614378:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2782 00000100:00080000:3.0:1789762855.614380:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762855.614384:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762855.614396:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762855/real 1789762855] req@ffff9018d070ca80 x1876699504262272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762855.614412:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504262272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762855.614420:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762855.614423:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762855.614427:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504262272:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762855.614457:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504262400:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789762855.614459:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762855.614474:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504262784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762855.614480:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762855.614490:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762855.614496:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762855/real 1789762855] req@ffff9018d070d180 x1876699504262400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762855.614505:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504262400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762855.614517:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762855.614520:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762855.614522:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504262400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762855.614526:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504262528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762855.614539:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762855.614546:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762855/real 1789762855] req@ffff9018d070e680 x1876699504262528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762855.614554:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504262528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762855.614560:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762855.614563:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762855.614565:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504262528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762855.614568:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504262656:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762855.614581:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762855.614585:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762855.614588:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762855.614594:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2782 00000100:00100000:0.0:1789762855.614595:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762855/real 1789762855] req@ffff9018d070e300 x1876699504262656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762855.614599:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762855.614608:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762855.614614:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762855.614616:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504262656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762855.614624:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762855.614626:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762855.614628:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504262656:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762855.614644:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504262784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762855.614665:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762855.614665:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504262912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762855.614677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762855/real 1789762855] req@ffff9018d070c000 x1876699504262784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762855.614686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504262784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762855.614692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762855.614694:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762855.614696:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504262784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762855.614702:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504262912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762855.614712:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762855.614718:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762855/real 1789762855] req@ffff9018d070dc00 x1876699504262912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762855.614724:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504262912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762855.614734:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762855.614735:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762855.614738:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504262912:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762856.254194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762857.278149:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762858.302156:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762859.326147:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762860.350191:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762860.734128:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762860.734139:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762860.734143:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762860.734152:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2787 00000100:00080000:3.0:1789762860.734163:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762860.734171:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762860.734207:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504263040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762860.734234:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762860.734239:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762860.734240:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762860.734244:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2787 00000100:00080000:3.0:1789762860.734247:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762860.734251:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762860.734261:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762860.734274:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504263168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762860.734281:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504263040:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762860.734288:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762860.734293:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762860.734295:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762860.734299:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2787 00000100:00080000:3.0:1789762860.734303:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762860.734307:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762860.734313:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762860.734326:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504263296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762860.734336:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762860.734340:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762860.734342:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762860.734347:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762860.734352:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2787 00000100:00080000:3.0:1789762860.734357:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762860.734362:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762860.734368:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762860.734382:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504263424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762860.734392:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762860/real 1789762860] req@ffff9018d070d500 x1876699504263040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762860.734395:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762860.734403:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762860.734407:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762860.734417:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2787 00000100:00080000:3.0:1789762860.734421:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762860.734426:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504263040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762860.734428:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762860.734436:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789762860.734446:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789762860.734452:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504263552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762860.734454:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762860.734463:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504263040:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762860.734472:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762860.734476:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762860.734478:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762860.734484:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2787 00000100:00080000:3.0:1789762860.734487:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762860.734491:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762860.734496:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504263168:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789762860.734499:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762860.734516:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504263680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762860.734529:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762860.734546:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762860/real 1789762860] req@ffff9018d070f800 x1876699504263168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762860.734560:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504263168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762860.734569:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762860.734582:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762860.734588:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504263168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762860.734594:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504263296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762860.734616:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762860.734627:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762860/real 1789762860] req@ffff9018d070f100 x1876699504263296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762860.734640:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504263296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762860.734649:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762860.734652:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762860.734656:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504263296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762860.734662:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504263424:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789762860.734682:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762860.734694:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762860/real 1789762860] req@ffff9018d070dc00 x1876699504263424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762860.734706:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504263424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762860.734714:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762860.734718:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762860.734721:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504263424:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789762860.734727:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504263552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762860.734746:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762860.734757:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762860/real 1789762860] req@ffff9018d070c000 x1876699504263552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762860.734769:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504263552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762860.734776:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762860.734780:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762860.734783:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504263552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762860.734802:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504263680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789762860.734822:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789762860.734834:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762860/real 1789762860] req@ffff9018d070e300 x1876699504263680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762860.734846:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504263680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762860.734855:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762860.734858:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762860.734862:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504263680:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762861.374157:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762862.398168:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762863.422196:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762864.446208:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762865.470202:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762865.854170:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762865.854180:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762865.854185:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762865.854195:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2792 00000100:00080000:0.0:1789762865.854208:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762865.854216:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762865.854260:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504263808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762865.854289:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762865.854292:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762865.854293:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762865.854296:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2792 00000100:00080000:0.0:1789762865.854298:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762865.854301:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762865.854311:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762865.854320:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504263936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762865.854328:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762865.854331:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762865.854332:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762865.854335:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2792 00000100:00080000:0.0:1789762865.854337:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762865.854339:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762865.854342:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762865.854347:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504263808:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762865.854355:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504264064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762865.854362:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762865.854365:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762865.854366:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762865.854369:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2792 00000100:00080000:0.0:1789762865.854371:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762865.854373:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762865.854376:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762865.854385:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504264192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762865.854392:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762865.854394:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762865.854395:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762865.854397:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2792 00000100:00080000:0.0:1789762865.854399:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762865.854402:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762865.854405:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762865.854412:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504264320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762865.854418:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762865.854421:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762865.854424:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762865.854425:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762865.854429:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2792 00000100:00080000:0.0:1789762865.854432:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762865.854434:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762865.854438:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762865.854447:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504264448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762865.854471:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762865/real 1789762865] req@ffff9018c8b9b100 x1876699504263808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762865.854492:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504263808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762865.854501:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762865.854504:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762865.854508:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504263808:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762865.854513:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504263936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762865.854530:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762865.854538:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762865/real 1789762865] req@ffff9018c8b9b480 x1876699504263936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762865.854546:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504263936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762865.854551:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762865.854553:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762865.854555:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504263936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762865.854576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504264064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762865.854588:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762865.854594:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762865/real 1789762865] req@ffff9018c8b98000 x1876699504264064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762865.854601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504264064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762865.854612:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762865.854614:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762865.854615:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504264064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762865.854619:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504264192:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762865.854629:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762865.854635:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762865/real 1789762865] req@ffff9018c8b9aa00 x1876699504264192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762865.854641:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504264192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762865.854647:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762865.854650:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762865.854652:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504264192:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762865.854656:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504264320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762865.854665:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762865.854671:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762865/real 1789762865] req@ffff9018c8b98e00 x1876699504264320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762865.854678:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504264320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762865.854683:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762865.854685:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762865.854688:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504264320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762865.854692:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504264448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762865.854702:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762865.854707:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762865/real 1789762865] req@ffff9018c8b9a680 x1876699504264448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762865.854714:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504264448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762865.854719:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762865.854721:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762865.854723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504264448:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789762866.494223:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762867.518178:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762868.542169:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762869.566192:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762870.590196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762870.974128:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762870.974138:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762870.974142:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762870.974153:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2797 00000100:00080000:1.0:1789762870.974165:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762870.974171:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762870.974216:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504264576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762870.974242:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762870.974246:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762870.974248:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762870.974251:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2797 00000100:00080000:1.0:1789762870.974254:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762870.974256:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762870.974266:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762870.974276:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504264704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762870.974284:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762870.974286:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762870.974288:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762870.974290:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2797 00000100:00080000:1.0:1789762870.974293:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762870.974298:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762870.974298:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504264576:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789762870.974305:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762870.974315:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504264832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762870.974321:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762870.974324:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762870.974326:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762870.974328:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2797 00000100:00080000:1.0:1789762870.974331:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762870.974333:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762870.974338:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762870.974346:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504264960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762870.974353:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762870.974355:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762870.974356:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762870.974358:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2797 00000100:00080000:1.0:1789762870.974361:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762870.974364:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762870.974367:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762870.974373:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504265088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762870.974381:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762870.974382:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762870.974385:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762870.974387:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762870.974391:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2797 00000100:00080000:1.0:1789762870.974393:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762870.974396:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762870.974399:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762870.974406:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504265216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762870.974438:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762870/real 1789762870] req@ffff9018c66a7800 x1876699504264576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762870.974460:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504264576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762870.974473:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762870.974478:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762870.974483:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504264576:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762870.974494:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504264704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762870.974515:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762870.974525:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762870/real 1789762870] req@ffff9018c66a7b80 x1876699504264704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762870.974544:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504264704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762870.974550:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762870.974553:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762870.974555:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504264704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762870.974577:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504264832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762870.974594:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762870.974605:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762870/real 1789762870] req@ffff9018c66a4000 x1876699504264832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762870.974617:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504264832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762870.974627:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762870.974630:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762870.974634:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504264832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762870.974640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504264960:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762870.974657:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762870.974665:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762870/real 1789762870] req@ffff9018c66a5f80 x1876699504264960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762870.974674:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504264960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762870.974680:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762870.974682:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762870.974684:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504264960:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762870.974687:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504265088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762870.974697:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762870.974702:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762870/real 1789762870] req@ffff9018c66a6d80 x1876699504265088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762870.974710:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504265088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762870.974715:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762870.974717:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762870.974718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504265088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762870.974722:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504265216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762870.974731:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762870.974736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762870/real 1789762870] req@ffff9018c66a5c00 x1876699504265216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762870.974743:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504265216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762870.974753:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762870.974755:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762870.974757:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504265216:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762871.614224:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762872.638191:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762873.662210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762874.686172:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762875.710255:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762876.094150:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762876.094158:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762876.094161:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762876.094169:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2802 00000100:00080000:3.0:1789762876.094178:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762876.094183:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762876.094219:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504265344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762876.094253:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762876.094256:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762876.094258:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762876.094261:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2802 00000100:00080000:3.0:1789762876.094264:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762876.094268:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762876.094276:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762876.094288:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504265472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762876.094301:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762876.094304:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762876.094307:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762876.094311:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2802 00000100:00080000:3.0:1789762876.094317:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762876.094330:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504265344:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762876.094330:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762876.094347:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762876.094363:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504265600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762876.094372:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762876.094374:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762876.094375:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762876.094378:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2802 00000100:00080000:3.0:1789762876.094381:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762876.094385:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762876.094388:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762876.094395:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504265728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762876.094406:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762876.094409:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762876.094410:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762876.094413:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2802 00000100:00080000:3.0:1789762876.094416:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762876.094418:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762876.094421:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762876.094428:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504265856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762876.094436:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762876.094446:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762876.094449:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762876.094451:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762876.094457:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2802 00000100:00080000:3.0:1789762876.094462:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762876.094467:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762876.094473:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762876.094485:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504265984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762876.094505:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762876/real 1789762876] req@ffff9018d070e680 x1876699504265344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762876.094528:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504265344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762876.094542:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762876.094548:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762876.094552:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504265344:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762876.094562:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504265472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762876.094591:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762876.094617:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762876/real 1789762876] req@ffff9018d070d180 x1876699504265472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762876.094627:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504265472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762876.094642:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762876.094646:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762876.094650:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504265472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762876.094682:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504265600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762876.094704:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762876.094715:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762876/real 1789762876] req@ffff9018d070ca80 x1876699504265600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762876.094727:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504265600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762876.094734:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762876.094737:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762876.094740:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504265600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762876.094747:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504265728:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762876.094766:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762876.094778:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762876/real 1789762876] req@ffff9018d070fb80 x1876699504265728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762876.094789:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504265728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762876.094797:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762876.094801:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762876.094805:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504265728:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762876.094812:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504265856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762876.094831:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762876.094840:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762876/real 1789762876] req@ffff9018d070ce00 x1876699504265856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762876.094849:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504265856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762876.094855:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762876.094858:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762876.094860:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504265856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762876.094864:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504265984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762876.094876:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762876.094881:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762876/real 1789762876] req@ffff9018d070df80 x1876699504265984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762876.094889:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504265984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762876.094894:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762876.094895:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762876.094897:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504265984:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762876.734168:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762877.758173:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762878.782242:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762879.806193:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762880.830177:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762881.214117:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762881.214125:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762881.214128:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762881.214137:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2807 00000100:00080000:0.0:1789762881.214145:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762881.214151:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762881.214188:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504266112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762881.214217:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762881.214220:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762881.214221:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762881.214225:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2807 00000100:00080000:0.0:1789762881.214228:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762881.214231:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762881.214239:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762881.214250:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504266240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762881.214260:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762881.214263:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762881.214265:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762881.214268:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2807 00000100:00080000:0.0:1789762881.214270:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762881.214273:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762881.214278:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762881.214290:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504266368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762881.214300:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504266112:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762881.214311:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762881.214317:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762881.214320:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762881.214326:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2807 00000100:00080000:0.0:1789762881.214329:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762881.214333:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762881.214338:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762881.214350:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504266496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762881.214359:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762881.214362:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762881.214363:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762881.214367:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2807 00000100:00080000:0.0:1789762881.214370:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762881.214372:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762881.214376:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762881.214385:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504266624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762881.214396:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762881.214397:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762881.214404:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762881.214405:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762881.214408:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2807 00000100:00080000:0.0:1789762881.214411:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762881.214414:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762881.214418:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762881.214430:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504266752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762881.214454:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762881/real 1789762881] req@ffff9018c8b98380 x1876699504266112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762881.214472:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504266112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762881.214482:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762881.214486:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762881.214490:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504266112:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762881.214496:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504266240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762881.214514:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762881.214522:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762881/real 1789762881] req@ffff9018c8b99c00 x1876699504266240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762881.214530:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504266240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762881.214536:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762881.214538:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762881.214540:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504266240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762881.214564:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504266368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762881.214578:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762881.214585:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762881/real 1789762881] req@ffff9018c8b99500 x1876699504266368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762881.214592:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504266368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762881.214598:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762881.214601:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762881.214604:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504266368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762881.214610:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504266496:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762881.214632:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762881.214644:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762881/real 1789762881] req@ffff9018c8b9ad80 x1876699504266496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762881.214657:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504266496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762881.214664:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762881.214667:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762881.214669:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504266496:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762881.214674:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504266624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762881.214689:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762881.214697:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762881/real 1789762881] req@ffff9018c8b9a300 x1876699504266624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762881.214709:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504266624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762881.214714:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762881.214716:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762881.214718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504266624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762881.214722:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504266752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762881.214733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762881.214739:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762881/real 1789762881] req@ffff9018c8b9a680 x1876699504266752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762881.214746:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504266752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762881.214751:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762881.214753:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762881.214755:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504266752:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762881.854175:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762882.878202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762883.902186:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762884.926197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762885.950152:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762886.334147:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762886.334156:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762886.334162:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762886.334172:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2812 00000100:00080000:1.0:1789762886.334184:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762886.334191:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762886.334236:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504266880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762886.334262:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762886.334265:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762886.334267:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762886.334270:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2812 00000100:00080000:1.0:1789762886.334272:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762886.334276:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762886.334285:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762886.334296:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504267008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762886.334303:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504266880:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762886.334306:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762886.334312:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762886.334313:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762886.334316:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2812 00000100:00080000:1.0:1789762886.334318:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762886.334320:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762886.334323:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762886.334333:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504267136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762886.334350:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762886.334354:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762886.334356:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762886.334361:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2812 00000100:00080000:1.0:1789762886.334366:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762886.334370:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762886.334377:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789762886.334378:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762886.334395:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504267264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762886.334403:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762886.334406:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762886.334408:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762886.334411:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2812 00000100:00080000:1.0:1789762886.334415:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762886.334419:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762886.334423:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762886/real 1789762886] req@ffff9018c66a5c00 x1876699504266880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762886.334425:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762886.334440:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504267392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762886.334447:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504266880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762886.334452:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762886.334461:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762886.334464:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762886.334468:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762886.334469:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2812 00000100:00080000:1.0:1789762886.334475:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762886.334477:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789762886.334480:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762886.334485:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504266880:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789762886.334486:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762886.334498:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504267520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762886.334515:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504267008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762886.334533:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762886.334542:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762886/real 1789762886] req@ffff9018c66a6d80 x1876699504267008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762886.334550:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504267008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762886.334557:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762886.334559:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762886.334561:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504267008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762886.334565:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504267136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762886.334576:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762886.334581:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762886/real 1789762886] req@ffff9018c66a5f80 x1876699504267136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762886.334589:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504267136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762886.334599:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762886.334601:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762886.334603:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504267136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762886.334607:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504267264:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762886.334617:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762886.334623:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762886/real 1789762886] req@ffff9018c66a4000 x1876699504267264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762886.334630:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504267264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762886.334644:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762886.334647:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762886.334649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504267264:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762886.334653:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504267392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762886.334666:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762886.334673:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762886/real 1789762886] req@ffff9018c66a7b80 x1876699504267392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762886.334681:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504267392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762886.334685:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762886.334687:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762886.334690:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504267392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762886.334694:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504267520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762886.334705:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762886.334715:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762886/real 1789762886] req@ffff9018c66a7800 x1876699504267520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762886.334727:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504267520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762886.334736:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762886.334739:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762886.334743:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504267520:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762886.974180:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762887.998161:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762889.022296:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762890.046247:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762891.070254:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762891.454143:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762891.454157:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762891.454163:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762891.454177:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2818 00000100:00080000:3.0:1789762891.454192:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762891.454201:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762891.454248:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504267648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762891.454296:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762891.454302:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762891.454304:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762891.454312:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2818 00000100:00080000:3.0:1789762891.454318:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762891.454327:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762891.454340:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762891.454343:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504267648:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762891.454362:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504267776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762891.454381:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762891.454386:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762891.454389:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762891.454396:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2818 00000100:00080000:3.0:1789762891.454403:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762891.454409:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762891.454417:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762891.454444:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504267904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762891.454458:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504267776:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762891.454566:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762891.454593:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504267904:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789762891.454696:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762891.454698:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762891.454703:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2818 00000100:00080000:3.0:1789762891.454707:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762891.454711:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762891.454716:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762891.454730:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504268032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762891.454745:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762891.454748:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762891.454750:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762891.454754:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2818 00000100:00080000:3.0:1789762891.454757:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762891.454762:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762891.454767:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762891.454777:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504268160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762891.454787:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504268032:192.168.203.158@tcp:8:kworker.0 00000100:00080000:3.0:1789762891.454793:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762891.454799:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762891.454802:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762891.454810:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2818 00000100:00080000:3.0:1789762891.454817:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762891.454827:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762891.454837:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504268160:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789762891.454838:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762891.454872:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504268288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762891.454931:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504268288:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762892.094249:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762893.118216:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762894.142199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762894.526419:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789762894.526477:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762894.526511:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762894.526557:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762894.526569:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762894.526578:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762894.526586:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762894.526598:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789762894.526620:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762891/real 1789762894] req@ffff9018d070ce00 x1876699504268160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762894.526653:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504268160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762894.526667:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762894.526673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762894.526678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504268160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762894.526687:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762891/real 1789762894] req@ffff9018d070df80 x1876699504268032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762894.526707:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504268032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762894.526716:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762894.526720:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762894.526724:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504268032:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762894.526731:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762891/real 1789762894] req@ffff9018d070d880 x1876699504267904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762894.526744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504267904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762894.526751:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762894.526755:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762894.526759:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504267904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762894.526767:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762891/real 1789762894] req@ffff9018d070ed80 x1876699504267776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762894.526780:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504267776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762894.526788:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762894.526792:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762894.526796:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504267776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762894.526803:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762891/real 1789762894] req@ffff9018d070f480 x1876699504267648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762894.526817:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504267648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762894.526825:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762894.526829:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762894.526832:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504267648:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762894.526889:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762891/real 1789762894] req@ffff9018d070fb80 x1876699504268288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762894.526904:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504268288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762894.526912:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762894.526916:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762894.526919:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504268288:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762895.166240:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762896.190304:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762896.574139:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762896.574163:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762896.574168:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762896.574180:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2823 00000100:00080000:3.0:1789762896.574194:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762896.574205:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762896.574253:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504268416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762896.574286:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762896.574291:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762896.574293:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762896.574298:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2823 00000100:00080000:3.0:1789762896.574302:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762896.574306:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762896.574320:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762896.574337:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504268544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762896.574359:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504268416:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762896.574361:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762896.574376:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762896.574381:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762896.574391:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2823 00000100:00080000:3.0:1789762896.574418:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762896.574429:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762896.574440:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762896.574466:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762896.574472:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504268672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762896.574496:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762896.574504:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762896.574508:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762896.574517:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2823 00000100:00080000:3.0:1789762896.574527:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762896.574537:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762896.574546:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762896/real 1789762896] req@ffff9018d070ca80 x1876699504268416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789762896.574554:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762896.574594:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504268800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762896.574620:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762896.574627:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504268416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762896.574630:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762896.574651:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762896.574654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762896.574666:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2823 00000100:00080000:1.0:1789762896.574673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762896.574684:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762896.574688:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504268416:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762896.574703:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762896.574717:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762896.574744:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504268544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762896.574749:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504268928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762896.574791:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762896.574799:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762896.574802:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762896.574811:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2823 00000100:00080000:3.0:1789762896.574821:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762896.574824:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762896.574835:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762896.574873:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762896.574893:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762896/real 1789762896] req@ffff9018d070d180 x1876699504268544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762896.574906:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504269056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762896.574943:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504268544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762896.574964:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762896.574972:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762896.574982:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504268544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762896.574999:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504268672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762896.575103:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762896.575133:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762896/real 1789762896] req@ffff9018d070e680 x1876699504268672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762896.575161:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504268672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762896.575174:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762896.575179:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762896.575184:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504268672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762896.575198:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504268800:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762896.575243:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762896.575269:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762896/real 1789762896] req@ffff9018d070e300 x1876699504268800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762896.575295:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504268800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762896.575312:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762896.575318:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762896.575325:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504268800:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762896.575364:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504268928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762896.575411:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762896.575438:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762896/real 1789762896] req@ffff9018d070c000 x1876699504268928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762896.575464:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504268928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762896.575483:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762896.575490:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762896.575498:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504268928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762896.575513:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504269056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762896.575555:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762896.575578:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762896/real 1789762896] req@ffff9018d070dc00 x1876699504269056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762896.575602:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504269056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762896.575633:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762896.575639:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762896.575646:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504269056:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762897.214251:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762898.238194:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762899.262209:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762900.286216:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762901.310251:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762901.694148:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762901.694160:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762901.694164:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762901.694194:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2828 00000100:00080000:3.0:1789762901.694211:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762901.694224:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762901.694282:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504269184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762901.694337:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762901.694342:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762901.694362:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762901.694367:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2828 00000100:00080000:3.0:1789762901.694372:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762901.694376:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762901.694387:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762901.694404:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504269312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762901.694418:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762901.694422:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762901.694425:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762901.694432:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504269184:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762901.694434:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2828 00000100:00080000:3.0:1789762901.694443:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762901.694449:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762901.694454:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762901.694468:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504269440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762901.694479:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762901.694483:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762901.694485:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762901.694489:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2828 00000100:00080000:3.0:1789762901.694493:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762901.694500:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762901.694507:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789762901.694516:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762901.694531:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504269568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762901.694550:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762901.694556:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762901.694560:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762901.694570:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2828 00000100:00100000:1.0:1789762901.694572:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762901/real 1789762901] req@ffff9018d070f100 x1876699504269184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762901.694577:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762901.694591:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762901.694606:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762901.694623:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504269184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762901.694631:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504269696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762901.694652:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762901.694660:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762901.694663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762901.694673:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789762901.694674:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504269184:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762901.694683:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762901.694694:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2828 00000100:00080000:3.0:1789762901.694701:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762901.694708:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762901.694712:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504269312:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789762901.694720:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762901.694742:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504269824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762901.694756:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762901.694800:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762901/real 1789762901] req@ffff9018d070f800 x1876699504269312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762901.694843:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504269312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762901.694860:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762901.694865:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762901.694870:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504269312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762901.694878:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504269440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762901.694924:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762901.694940:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762901/real 1789762901] req@ffff9018d070d500 x1876699504269440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762901.694955:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504269440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762901.694964:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762901.694968:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762901.694972:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504269440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762901.694979:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504269568:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762901.694997:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762901.695008:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762901/real 1789762901] req@ffff9018d070dc00 x1876699504269568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762901.695021:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504269568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762901.695031:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762901.695034:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762901.695037:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504269568:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762901.695043:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504269696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762901.695136:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762901.695155:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762901/real 1789762901] req@ffff9018d070c000 x1876699504269696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762901.695180:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504269696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762901.695196:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762901.695202:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762901.695209:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504269696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762901.695245:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504269824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762901.695283:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762901.695303:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762901/real 1789762901] req@ffff9018d070e300 x1876699504269824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762901.695327:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504269824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762901.695359:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762901.695365:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762901.695370:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504269824:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762902.334209:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762903.358148:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762904.382224:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762905.406222:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762906.430297:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762906.814183:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762906.814193:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762906.814198:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762906.814208:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2833 00000100:00080000:1.0:1789762906.814220:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762906.814228:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762906.814281:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504269952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762906.814317:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762906.814320:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762906.814322:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762906.814326:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2833 00000100:00080000:1.0:1789762906.814328:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762906.814332:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762906.814343:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762906.814356:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504270080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762906.814366:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762906.814370:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762906.814371:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762906.814376:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2833 00000100:00080000:1.0:1789762906.814384:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762906.814385:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504269952:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762906.814388:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762906.814398:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762906.814408:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504270208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762906.814418:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762906.814422:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762906.814424:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762906.814427:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2833 00000100:00080000:1.0:1789762906.814430:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762906.814435:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762906.814439:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762906.814447:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504270336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762906.814454:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762906.814458:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762906.814460:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762906.814467:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762906.814473:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2833 00000100:00080000:1.0:1789762906.814478:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762906.814482:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762906.814487:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762906.814495:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504270464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762906.814503:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762906.814507:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762906.814509:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762906.814512:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2833 00000100:00080000:1.0:1789762906.814518:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762906.814518:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762906/real 1789762906] req@ffff9018c66a5180 x1876699504269952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762906.814529:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762906.814540:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762906.814564:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504270592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762906.814565:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504269952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762906.814630:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762906.814640:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762906.814648:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504269952:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762906.814671:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504270080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762906.814687:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762906.814696:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762906/real 1789762906] req@ffff9018c66a6300 x1876699504270080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762906.814709:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504270080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762906.814714:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762906.814716:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762906.814718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504270080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762906.814722:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504270208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762906.814733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762906.814739:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762906/real 1789762906] req@ffff9018c66a4700 x1876699504270208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762906.814746:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504270208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762906.814751:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762906.814753:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762906.814755:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504270208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762906.814758:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504270336:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762906.814787:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762906.814796:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762906/real 1789762906] req@ffff9018c66a6680 x1876699504270336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762906.814803:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504270336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762906.814813:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762906.814815:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762906.814817:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504270336:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762906.814821:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504270464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762906.814834:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762906.814840:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762906/real 1789762906] req@ffff9018c66a5500 x1876699504270464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762906.814847:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504270464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762906.814852:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762906.814854:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762906.814856:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504270464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762906.814859:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504270592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762906.814870:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762906.814875:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762906/real 1789762906] req@ffff9018c66a4e00 x1876699504270592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762906.814883:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504270592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762906.814887:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762906.814889:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762906.814891:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504270592:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762907.454253:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762908.478227:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762909.502203:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762910.526195:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762911.550178:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762911.934183:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762911.934193:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762911.934197:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762911.934208:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2838 00000100:00080000:3.0:1789762911.934220:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762911.934227:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762911.934272:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504270720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762911.934300:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762911.934303:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762911.934304:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762911.934308:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2838 00000100:00080000:3.0:1789762911.934310:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762911.934313:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762911.934323:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762911.934336:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504270848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762911.934349:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762911.934353:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504270720:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762911.934354:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762911.934357:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762911.934369:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2838 00000100:00080000:3.0:1789762911.934373:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762911.934379:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762911.934386:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762911.934406:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504270976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762911.934414:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762911.934417:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762911.934419:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762911.934424:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2838 00000100:00080000:3.0:1789762911.934429:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762911.934435:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762911.934442:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762911.934450:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762911.934462:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504271104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762911.934471:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762911.934474:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762911.934475:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762911.934479:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2838 00000100:00100000:2.0:1789762911.934485:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762911/real 1789762911] req@ffff9018d070e680 x1876699504270720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762911.934485:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762911.934500:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762911.934514:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762911.934527:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504270720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762911.934529:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504271232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762911.934586:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762911.934592:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762911.934594:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762911.934598:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2838 00000100:00080000:2.0:1789762911.934599:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762911.934600:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762911.934608:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762911.934614:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762911.934615:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762911.934632:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504270720:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762911.934638:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504271360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762911.934675:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504270848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762911.934698:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762911.934708:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762911/real 1789762911] req@ffff9018d070d180 x1876699504270848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762911.934717:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504270848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762911.934723:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762911.934725:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762911.934727:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504270848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762911.934749:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504270976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762911.934764:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762911.934772:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762911/real 1789762911] req@ffff9018d070ca80 x1876699504270976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762911.934780:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504270976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762911.934785:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762911.934787:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762911.934789:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504270976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762911.934794:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504271104:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762911.934805:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762911.934810:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762911/real 1789762911] req@ffff9018d070fb80 x1876699504271104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762911.934818:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504271104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762911.934823:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762911.934825:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762911.934826:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504271104:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762911.934830:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504271232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762911.934840:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762911.934846:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762911/real 1789762911] req@ffff9018d070f480 x1876699504271232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762911.934853:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504271232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762911.934858:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762911.934860:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762911.934862:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504271232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762911.934865:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504271360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762911.934875:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762911.934880:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762911/real 1789762911] req@ffff9018d070ed80 x1876699504271360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762911.934887:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504271360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762911.934892:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762911.934894:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762911.934896:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504271360:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762912.574228:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762913.598222:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762914.622251:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762915.646252:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762916.670222:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762917.054216:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762917.054228:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762917.054233:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762917.054243:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2843 00000100:00080000:1.0:1789762917.054256:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762917.054264:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762917.054312:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504271488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762917.054345:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762917.054348:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762917.054350:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762917.054353:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2843 00000100:00080000:1.0:1789762917.054356:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762917.054358:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762917.054368:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762917.054378:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504271616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762917.054386:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762917.054389:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762917.054390:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762917.054393:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2843 00000100:00080000:1.0:1789762917.054395:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762917.054397:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762917.054403:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762917.054405:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504271488:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762917.054414:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504271744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762917.054422:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762917.054425:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762917.054426:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762917.054429:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2843 00000100:00080000:1.0:1789762917.054430:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762917.054433:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762917.054436:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762917.054444:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504271872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762917.054450:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762917.054452:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762917.054453:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762917.054456:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2843 00000100:00080000:1.0:1789762917.054459:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762917.054462:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762917.054465:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762917.054472:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504272000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762917.054481:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762917.054486:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1789762917.054488:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762917.054491:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762917.054499:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2843 00000100:00080000:1.0:1789762917.054504:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762917.054509:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762917.054517:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762917.054536:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504272128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762917.054537:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762917/real 1789762917] req@ffff9018c66a4380 x1876699504271488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762917.054579:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504271488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762917.054589:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762917.054593:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762917.054597:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504271488:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762917.054612:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504271616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762917.054629:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762917.054639:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762917/real 1789762917] req@ffff9018c66a6a00 x1876699504271616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762917.054648:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504271616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762917.054654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762917.054656:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762917.054658:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504271616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762917.054685:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504271744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762917.054716:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762917.054725:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762917/real 1789762917] req@ffff9018c66a7480 x1876699504271744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762917.054748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504271744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762917.054764:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762917.054768:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762917.054771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504271744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762917.054778:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504271872:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762917.054796:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762917.054804:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762917/real 1789762917] req@ffff9018c66a4a80 x1876699504271872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762917.054812:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504271872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762917.054818:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762917.054820:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762917.054822:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504271872:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762917.054825:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504272000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762917.054836:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762917.054842:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762917/real 1789762917] req@ffff9018c66a7100 x1876699504272000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762917.054850:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504272000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762917.054855:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762917.054856:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762917.054859:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504272000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762917.054863:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504272128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762917.054873:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762917.054878:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762917/real 1789762917] req@ffff9018c66a5880 x1876699504272128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762917.054885:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504272128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762917.054891:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762917.054893:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762917.054894:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504272128:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762917.694268:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762918.718251:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762919.742201:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762920.766212:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762921.790197:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762922.174136:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762922.174146:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762922.174150:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762922.174161:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2848 00000100:00080000:1.0:1789762922.174173:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762922.174180:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762922.174226:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504272256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762922.174252:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762922.174255:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762922.174257:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762922.174260:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2848 00000100:00080000:1.0:1789762922.174263:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762922.174266:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762922.174275:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762922.174284:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504272384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762922.174296:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762922.174298:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762922.174300:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762922.174302:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2848 00000100:00080000:1.0:1789762922.174304:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762922.174307:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762922.174310:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762922.174316:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504272512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762922.174323:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762922.174325:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762922.174326:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762922.174331:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2848 00000100:00100000:2.0:1789762922.174333:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504272256:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762922.174334:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762922.174340:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762922.174345:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762922.174353:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504272640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762922.174360:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762922.174363:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762922.174364:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762922.174367:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2848 00000100:00080000:1.0:1789762922.174370:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762922.174373:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762922.174377:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762922.174384:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504272768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762922.174390:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762922.174392:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762922.174394:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762922.174397:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2848 00000100:00080000:1.0:1789762922.174399:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762922.174401:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762922.174404:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762922.174410:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762922.174415:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504272896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762922.174458:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762922/real 1789762922] req@ffff9018c66a5880 x1876699504272256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762922.174473:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504272256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762922.174482:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762922.174485:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762922.174489:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504272256:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762922.174495:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504272384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762922.174510:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762922.174519:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762922/real 1789762922] req@ffff9018c66a7100 x1876699504272384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762922.174527:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504272384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762922.174532:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762922.174534:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762922.174536:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504272384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762922.174562:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504272512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762922.174573:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762922.174579:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762922/real 1789762922] req@ffff9018c66a4a80 x1876699504272512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762922.174587:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504272512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762922.174592:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762922.174593:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762922.174595:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504272512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762922.174599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504272640:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762922.174609:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762922.174615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762922/real 1789762922] req@ffff9018c66a7480 x1876699504272640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762922.174623:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504272640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762922.174629:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762922.174630:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762922.174632:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504272640:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762922.174635:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504272768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762922.174646:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762922.174652:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762922/real 1789762922] req@ffff9018c66a6a00 x1876699504272768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762922.174680:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504272768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762922.174686:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762922.174690:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762922.174694:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504272768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762922.174699:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504272896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762922.174720:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762922.174732:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762922/real 1789762922] req@ffff9018c66a4380 x1876699504272896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762922.174745:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504272896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762922.174758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762922.174760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762922.174762:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504272896:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762922.814251:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762923.838219:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762924.862179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762925.886236:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762926.910181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762927.294134:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762927.294148:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762927.294154:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762927.294166:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2853 00000100:00080000:3.0:1789762927.294180:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762927.294189:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762927.294238:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504273024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762927.294267:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762927.294272:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762927.294274:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762927.294279:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2853 00000100:00080000:3.0:1789762927.294283:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762927.294288:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762927.294301:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762927.294317:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504273024:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762927.294318:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504273152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762927.294336:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762927.294339:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762927.294342:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762927.294347:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2853 00000100:00080000:3.0:1789762927.294351:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762927.294355:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762927.294360:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762927.294372:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504273280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762927.294382:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762927.294385:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762927.294388:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762927.294392:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2853 00000100:00080000:3.0:1789762927.294397:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762927.294401:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762927.294404:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762927.294411:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762927.294426:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504273408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762927.294437:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762927.294440:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762927.294443:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762927.294447:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2853 00000100:00080000:3.0:1789762927.294452:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762927.294456:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762927/real 1789762927] req@ffff9018d070d880 x1876699504273024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762927.294461:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762927.294471:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762927.294481:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504273024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762927.294487:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504273536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762927.294492:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762927.294499:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762927.294507:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504273024:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762927.294507:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762927.294518:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762927.294520:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762927.294526:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2853 00000100:00080000:3.0:1789762927.294530:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762927.294534:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762927.294538:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504273152:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789762927.294542:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762927.294559:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504273664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762927.294563:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762927.294577:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762927/real 1789762927] req@ffff9018d070df80 x1876699504273152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762927.294587:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504273152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762927.294599:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762927.294601:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762927.294604:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504273152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762927.294608:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504273280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762927.294637:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762927.294646:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762927/real 1789762927] req@ffff9018d070ce00 x1876699504273280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762927.294654:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504273280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762927.294661:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762927.294663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762927.294665:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504273280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762927.294669:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504273408:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762927.294684:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762927.294690:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762927/real 1789762927] req@ffff9018d070ed80 x1876699504273408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762927.294698:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504273408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762927.294703:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762927.294705:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762927.294708:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504273408:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762927.294711:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504273536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762927.294722:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762927.294728:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762927/real 1789762927] req@ffff9018d070f480 x1876699504273536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762927.294735:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504273536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762927.294740:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762927.294742:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762927.294746:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504273536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762927.294769:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504273664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762927.294793:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762927.294805:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762927/real 1789762927] req@ffff9018d070fb80 x1876699504273664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762927.294815:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504273664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762927.294823:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762927.294828:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762927.294832:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504273664:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762927.934255:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762928.958196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762929.982221:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762931.006321:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762932.030321:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762932.414175:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762932.414208:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762932.414214:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762932.414231:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2858 00000100:00080000:3.0:1789762932.414278:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762932.414291:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762932.414350:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504273792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762932.414383:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762932.414388:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762932.414391:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762932.414396:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2858 00000100:00080000:3.0:1789762932.414402:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762932.414408:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762932.414421:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762932.414439:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504273920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762932.414441:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504273792:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762932.414463:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762932.414466:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762932.414468:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762932.414472:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2858 00000100:00080000:3.0:1789762932.414477:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762932.414481:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762932.414486:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762932.414501:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504274048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762932.414512:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762932.414532:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762932.414534:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762932.414537:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2858 00000100:00080000:3.0:1789762932.414550:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762932.414554:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762932.414560:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789762932.414568:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762932.414575:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504274176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762932.414617:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762932.414638:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762932.414640:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762932.414646:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2858 00000100:00080000:3.0:1789762932.414652:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762932.414657:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762932.414664:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762932.414667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762932/real 1789762932] req@ffff9018d070ca80 x1876699504273792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762932.414682:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504274304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762932.414695:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762932.414696:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504273792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762932.414704:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762932.414710:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762932.414722:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2858 00000100:00080000:2.0:1789762932.414725:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762932.414727:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762932.414737:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762932.414738:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762932.414749:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762932.414752:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504273792:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762932.414763:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504274432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762932.414790:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504273920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762932.414839:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762932.414852:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762932/real 1789762932] req@ffff9018d070d180 x1876699504273920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762932.414864:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504273920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762932.414872:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762932.414876:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762932.414879:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504273920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762932.414884:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504274048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762932.414903:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762932.414918:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762932/real 1789762932] req@ffff9018d070e680 x1876699504274048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762932.414936:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504274048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762932.414948:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762932.414953:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762932.414958:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504274048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762932.414967:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504274176:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762932.414993:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762932.415005:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762932/real 1789762932] req@ffff9018d070e300 x1876699504274176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762932.415016:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504274176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762932.415024:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762932.415027:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762932.415029:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504274176:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762932.415034:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504274304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762932.415048:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762932.415057:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762932/real 1789762932] req@ffff9018d070c000 x1876699504274304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762932.415091:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504274304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762932.415105:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762932.415108:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762932.415111:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504274304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762932.415116:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504274432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762932.415134:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762932.415144:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762932/real 1789762932] req@ffff9018d070dc00 x1876699504274432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762932.415155:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504274432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762932.415166:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762932.415171:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762932.415175:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504274432:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762933.054226:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762934.078240:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762935.102286:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762936.126158:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762937.150141:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762937.534112:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762937.534124:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762937.534128:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762937.534138:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2864 00000100:00080000:1.0:1789762937.534149:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762937.534156:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762937.534193:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504274560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762937.534221:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762937.534226:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762937.534228:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762937.534232:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2864 00000100:00080000:1.0:1789762937.534237:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762937.534241:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762937.534252:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762937.534265:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504274688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762937.534277:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762937.534281:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762937.534283:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762937.534289:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504274560:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762937.534291:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2864 00000100:00080000:1.0:1789762937.534299:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762937.534304:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762937.534310:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762937.534326:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504274816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762937.534338:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762937.534342:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762937.534344:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762937.534350:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2864 00000800:00000100:2.0:1789762937.534350:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762937.534359:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762937.534364:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762937.534372:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762937.534383:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762937/real 1789762937] req@ffff9018c66a4e00 x1876699504274560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762937.534394:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504274944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762937.534401:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504274560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762937.534416:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762937.534423:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762937.534425:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762937.534426:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762937.534434:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789762937.534436:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2864 00000100:00080000:1.0:1789762937.534442:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762937.534442:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504274560:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789762937.534449:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762937.534453:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504274688:192.168.203.158@tcp:38:kworker.0 00000008:00100000:1.0:1789762937.534457:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762937.534472:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504275072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762937.534479:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762937.534482:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789762937.534483:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762937.534487:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789762937.534491:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2864 00000100:00080000:1.0:1789762937.534496:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762937.534499:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789762937.534502:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762937.534503:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762937/real 1789762937] req@ffff9018c66a5500 x1876699504274688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762937.534514:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504275200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762937.534517:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504274688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762937.534539:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762937.534626:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762937.534629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504274688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762937.534650:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504274816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762937.534664:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762937.534671:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762937/real 1789762937] req@ffff9018c66a6680 x1876699504274816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762937.534678:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504274816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762937.534684:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762937.534686:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762937.534688:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504274816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762937.534692:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504274944:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762937.534702:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762937.534707:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762937/real 1789762937] req@ffff9018c66a4700 x1876699504274944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762937.534714:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504274944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762937.534720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762937.534721:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762937.534723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504274944:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762937.534726:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504275072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762937.534736:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762937.534742:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762937/real 1789762937] req@ffff9018c66a6300 x1876699504275072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762937.534748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504275072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762937.534753:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762937.534755:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762937.534757:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504275072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762937.534760:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504275200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762937.534769:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762937.534774:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762937/real 1789762937] req@ffff9018c66a5180 x1876699504275200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762937.534781:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504275200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762937.534786:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762937.534788:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762937.534790:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504275200:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789762938.174218:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762939.198183:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762940.222223:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762941.246243:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762942.270174:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762942.654111:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762942.654118:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762942.654122:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762942.654131:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2869 00000100:00080000:0.0:1789762942.654140:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762942.654145:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762942.654175:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504275328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762942.654196:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762942.654202:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762942.654203:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762942.654206:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2869 00000100:00080000:0.0:1789762942.654209:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762942.654211:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762942.654219:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762942.654227:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504275456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762942.654234:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762942.654236:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762942.654237:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762942.654239:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2869 00000100:00080000:0.0:1789762942.654241:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762942.654243:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762942.654246:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762942.654252:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504275584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762942.654258:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762942.654260:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762942.654267:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762942.654270:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2869 00000100:00080000:0.0:1789762942.654275:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762942.654278:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762942.654281:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762942.654291:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504275712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762942.654307:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504275328:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762942.654309:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762942.654327:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762942.654329:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762942.654333:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2869 00000100:00080000:0.0:1789762942.654335:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762942.654338:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762942.654342:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762942.654350:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504275840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762942.654357:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762942.654360:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762942.654361:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762942.654364:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2869 00000100:00080000:0.0:1789762942.654366:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762942.654370:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762942.654373:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762942.654379:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504275968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762942.654408:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762942.654464:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762942/real 1789762942] req@ffff9018c8b98e00 x1876699504275328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762942.654486:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504275328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762942.654498:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762942.654502:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762942.654517:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504275328:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762942.654525:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504275456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762942.654546:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762942.654557:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762942/real 1789762942] req@ffff9018c8b9aa00 x1876699504275456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762942.654571:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504275456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762942.654577:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762942.654579:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762942.654581:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504275456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762942.654586:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504275584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762942.654597:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762942.654603:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762942/real 1789762942] req@ffff9018c8b98000 x1876699504275584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762942.654611:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504275584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762942.654623:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762942.654625:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762942.654627:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504275584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762942.654654:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504275712:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762942.654667:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762942.654674:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762942/real 1789762942] req@ffff9018c8b9b480 x1876699504275712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762942.654682:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504275712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762942.654692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762942.654694:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762942.654696:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504275712:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762942.654700:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504275840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762942.654710:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762942.654716:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762942/real 1789762942] req@ffff9018c8b9b100 x1876699504275840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762942.654723:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504275840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762942.654728:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762942.654730:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762942.654732:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504275840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762942.654735:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504275968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762942.654745:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762942.654751:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762942/real 1789762942] req@ffff9018c8b9b800 x1876699504275968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762942.654758:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504275968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762942.654763:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762942.654764:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762942.654767:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504275968:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762943.294239:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762944.318153:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762945.342152:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762946.366267:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762947.390177:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762947.774097:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762947.774105:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762947.774108:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762947.774116:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2874 00000100:00080000:0.0:1789762947.774126:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762947.774131:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762947.774161:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504276096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762947.774184:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762947.774187:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762947.774188:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762947.774191:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2874 00000100:00080000:0.0:1789762947.774193:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762947.774195:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762947.774203:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762947.774210:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504276224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762947.774217:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762947.774219:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762947.774220:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762947.774223:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2874 00000100:00080000:0.0:1789762947.774225:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762947.774227:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762947.774229:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762947.774235:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504276352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762947.774241:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762947.774244:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762947.774245:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762947.774248:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2874 00000100:00080000:0.0:1789762947.774252:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762947.774254:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762947.774257:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762947.774263:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504276480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762947.774272:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762947.774275:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762947.774277:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762947.774284:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504276096:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762947.774285:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2874 00000100:00080000:0.0:1789762947.774294:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762947.774308:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762947.774312:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762947.774324:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504276608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762947.774331:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762947.774334:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762947.774335:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762947.774338:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2874 00000100:00080000:0.0:1789762947.774340:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762947.774342:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762947.774346:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762947.774356:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504276736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762947.774386:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762947.774441:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762947/real 1789762947] req@ffff9018c8b9bb80 x1876699504276096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762947.774457:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504276096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762947.774483:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762947.774489:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762947.774494:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504276096:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762947.774502:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504276224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762947.774530:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762947.774542:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762947/real 1789762947] req@ffff9018c8b98700 x1876699504276224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762947.774551:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504276224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762947.774557:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762947.774560:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762947.774563:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504276224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762947.774568:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504276352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762947.774582:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762947.774589:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762947/real 1789762947] req@ffff9018c8b99f80 x1876699504276352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762947.774597:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504276352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762947.774602:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762947.774604:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762947.774605:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504276352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762947.774631:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504276480:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789762947.774643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762947.774649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762947/real 1789762947] req@ffff9018c8b98a80 x1876699504276480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762947.774657:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504276480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762947.774662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762947.774664:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762947.774666:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504276480:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762947.774669:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504276608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762947.774680:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762947.774686:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762947/real 1789762947] req@ffff9018c8b99880 x1876699504276608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762947.774693:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504276608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762947.774698:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762947.774700:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762947.774702:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504276608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762947.774705:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504276736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762947.774716:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762947.774722:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762947/real 1789762947] req@ffff9018c8b9b800 x1876699504276736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762947.774728:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504276736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762947.774733:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762947.774735:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762947.774737:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504276736:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762948.414182:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762949.438212:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762950.462179:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762951.486232:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762952.510224:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762952.894172:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762952.894182:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762952.894187:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762952.894197:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2879 00000100:00080000:2.0:1789762952.894207:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762952.894214:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762952.894252:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504276864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762952.894283:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762952.894287:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762952.894289:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762952.894293:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2879 00000100:00080000:2.0:1789762952.894296:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762952.894300:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762952.894310:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762952.894323:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504276992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762952.894334:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762952.894339:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762952.894344:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762952.894345:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504276864:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762952.894354:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2879 00000100:00080000:2.0:1789762952.894362:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762952.894368:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762952.894376:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762952.894398:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504277120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762952.894425:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762952.894425:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762952.894453:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762952.894456:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762952.894463:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2879 00000100:00080000:2.0:1789762952.894469:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762952.894474:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762952.894481:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762952.894500:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504277248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762952.894509:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762952/real 1789762952] req@ffff9018d0f24380 x1876699504276864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762952.894517:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762952.894528:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762952.894532:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762952.894539:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2879 00000100:00080000:2.0:1789762952.894544:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762952.894548:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504276864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762952.894556:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762952.894571:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762952.894577:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762952.894589:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762952.894596:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504277376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762952.894604:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504276864:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789762952.894627:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762952.894631:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762952.894634:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762952.894639:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2879 00000100:00080000:2.0:1789762952.894645:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762952.894651:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504276992:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789762952.894652:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762952.894661:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762952.894674:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504277504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762952.894732:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762952.894758:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762952/real 1789762952] req@ffff9018d0f25180 x1876699504276992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762952.894786:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504276992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762952.894803:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762952.894810:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762952.894817:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504276992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762952.894831:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504277120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762952.894873:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762952.894896:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762952/real 1789762952] req@ffff9018d0f27800 x1876699504277120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762952.894923:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504277120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762952.894953:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762952.894960:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762952.894967:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504277120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762952.894979:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504277248:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762952.895025:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762952.895119:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762952/real 1789762952] req@ffff9018d0f25c00 x1876699504277248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762952.895145:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504277248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762952.895163:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762952.895170:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762952.895177:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504277248:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762952.895190:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504277376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762952.895233:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762952.895254:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762952/real 1789762952] req@ffff9018d0f26a00 x1876699504277376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762952.895270:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504277376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762952.895281:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762952.895284:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762952.895289:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504277376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762952.895314:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504277504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762952.895340:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762952.895353:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762952/real 1789762952] req@ffff9018d0f24a80 x1876699504277504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762952.895367:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504277504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762952.895393:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762952.895395:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762952.895398:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504277504:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762953.534173:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762954.558220:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762955.582139:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762956.606162:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762957.630188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762958.014152:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762958.014163:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762958.014168:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762958.014180:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2884 00000100:00080000:3.0:1789762958.014192:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762958.014200:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762958.014244:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504277632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762958.014269:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762958.014273:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762958.014274:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762958.014277:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2884 00000100:00080000:3.0:1789762958.014279:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762958.014282:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762958.014291:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762958.014300:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504277760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762958.014307:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762958.014309:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762958.014310:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762958.014313:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2884 00000100:00080000:3.0:1789762958.014315:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762958.014318:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762958.014322:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762958.014328:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504277888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762958.014339:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762958.014342:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762958.014345:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762958.014348:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2884 00000100:00080000:3.0:1789762958.014350:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762958.014352:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762958.014356:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762958.014365:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504277632:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762958.014369:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504278016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762958.014384:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762958.014387:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762958.014388:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762958.014407:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2884 00000100:00080000:3.0:1789762958.014411:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762958.014414:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762958.014420:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762958.014431:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504278144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762958.014438:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762958.014441:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762958.014443:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762958.014446:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2884 00000100:00080000:3.0:1789762958.014449:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762958.014453:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762958.014456:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762958.014464:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504278272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762958.014496:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504277760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762958.014522:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504277888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762958.014549:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504278016:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762958.014568:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504278144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762958.014582:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504278272:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762958.654370:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762959.678155:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762960.702187:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762961.086389:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789762961.086442:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762961.086460:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762961.086514:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762961.086525:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762961.086536:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762961.086548:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789762961.086559:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789762961.086581:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762958/real 1789762961] req@ffff9018d070dc00 x1876699504278016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762961.086616:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504278016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762961.086635:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762961.086641:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762961.086644:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504278016:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762961.086654:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762958/real 1789762961] req@ffff9018d070c000 x1876699504278144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762961.086676:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504278144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762961.086686:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762961.086690:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762961.086694:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504278144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762961.086701:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762958/real 1789762961] req@ffff9018d070e300 x1876699504278272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762961.086713:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504278272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762961.086721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762961.086725:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762961.086729:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504278272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762961.086736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762958/real 1789762961] req@ffff9018d070d500 x1876699504277632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762961.086754:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504277632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762961.086763:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762961.086767:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762961.086770:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504277632:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762961.086778:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762958/real 1789762961] req@ffff9018d070f800 x1876699504277760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762961.086790:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504277760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762961.086799:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762961.086802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762961.086805:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504277760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762961.086812:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762958/real 1789762961] req@ffff9018d070f100 x1876699504277888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762961.086824:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504277888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762961.086834:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762961.086838:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762961.086856:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504277888:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762961.726169:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762962.750237:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762963.134159:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762963.134172:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762963.134178:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762963.134190:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2889 00000100:00080000:3.0:1789762963.134204:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762963.134213:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762963.134261:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504278400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762963.134295:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762963.134300:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762963.134302:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762963.134308:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2889 00000100:00080000:3.0:1789762963.134313:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762963.134317:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762963.134329:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762963.134344:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504278528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762963.134369:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762963.134374:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762963.134377:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762963.134382:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2889 00000100:00080000:3.0:1789762963.134386:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762963.134392:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762963.134398:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762963.134403:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504278400:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762963.134416:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504278656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762963.134431:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762963.134436:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762963.134438:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762963.134453:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2889 00000100:00080000:3.0:1789762963.134459:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762963.134464:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762963.134471:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762963.134485:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504278784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762963.134497:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789762963.134499:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762963.134506:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762963.134508:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762963.134514:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2889 00000100:00080000:3.0:1789762963.134519:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762963.134525:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762963.134535:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762963.134548:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504278912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762963.134559:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762963/real 1789762963] req@ffff9018d070e680 x1876699504278400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762963.134560:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762963.134569:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762963.134573:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762963.134584:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2889 00000100:00080000:3.0:1789762963.134590:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762963.134596:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762963.134602:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504278400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789762963.134602:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762963.134619:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504279040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762963.134620:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762963.134631:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762963.134638:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504278400:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762963.134648:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504278528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762963.134677:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762963.134692:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762963/real 1789762963] req@ffff9018d070d180 x1876699504278528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762963.134707:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504278528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762963.134716:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762963.134719:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762963.134723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504278528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762963.134754:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504278656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762963.134778:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762963.134791:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762963/real 1789762963] req@ffff9018d070ca80 x1876699504278656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762963.134804:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504278656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762963.134820:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762963.134824:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762963.134828:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504278656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762963.134834:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504278784:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762963.134857:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762963.134868:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762963/real 1789762963] req@ffff9018d070fb80 x1876699504278784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762963.134879:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504278784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762963.134888:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762963.134891:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762963.134894:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504278784:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762963.134900:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504278912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762963.134921:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762963.134931:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762963/real 1789762963] req@ffff9018d070f480 x1876699504278912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762963.134942:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504278912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762963.134949:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762963.134952:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762963.134956:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504278912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762963.134963:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504279040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762963.134982:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762963.134993:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762963/real 1789762963] req@ffff9018d070ed80 x1876699504279040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762963.135005:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504279040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762963.135013:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762963.135017:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762963.135020:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504279040:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762963.774146:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762964.798224:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762965.822161:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762966.846187:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762967.870265:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762968.254225:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762968.254236:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762968.254241:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762968.254264:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2894 00000100:00080000:2.0:1789762968.254277:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762968.254286:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762968.254357:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504279168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762968.254395:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762968.254398:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762968.254400:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762968.254404:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2894 00000100:00080000:2.0:1789762968.254407:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762968.254410:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762968.254420:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762968.254431:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504279296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762968.254439:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762968.254442:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762968.254443:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762968.254446:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2894 00000100:00080000:2.0:1789762968.254447:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762968.254452:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789762968.254455:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504279168:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789762968.254456:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762968.254470:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504279424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762968.254476:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762968.254479:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762968.254480:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762968.254482:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2894 00000100:00080000:2.0:1789762968.254485:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762968.254487:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762968.254491:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762968.254498:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504279552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762968.254505:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762968.254507:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762968.254508:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762968.254511:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2894 00000100:00080000:2.0:1789762968.254515:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762968.254519:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762968.254524:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762968.254538:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504279680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762968.254547:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789762968.254555:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762968.254560:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762968.254562:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762968.254569:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2894 00000100:00080000:2.0:1789762968.254583:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762968.254587:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762968.254602:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762968.254615:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504279808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762968.254624:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762968/real 1789762968] req@ffff9018d0f26d80 x1876699504279168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762968.254650:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504279168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762968.254665:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762968.254682:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762968.254686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504279168:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762968.254692:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504279296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762968.254714:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762968.254726:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762968/real 1789762968] req@ffff9018d0f25f80 x1876699504279296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762968.254735:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504279296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762968.254741:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762968.254744:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762968.254746:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504279296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762968.254773:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504279424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762968.254785:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762968.254792:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762968/real 1789762968] req@ffff9018d0f27100 x1876699504279424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762968.254800:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504279424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762968.254805:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762968.254807:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762968.254809:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504279424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762968.254813:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504279552:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762968.254825:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762968.254839:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762968/real 1789762968] req@ffff9018d0f26300 x1876699504279552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762968.254847:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504279552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762968.254854:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762968.254857:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762968.254859:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504279552:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762968.254863:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504279680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762968.254876:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762968.254882:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762968/real 1789762968] req@ffff9018d0f24a80 x1876699504279680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762968.254890:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504279680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762968.254895:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762968.254897:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762968.254899:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504279680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762968.254902:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504279808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762968.254912:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762968.254919:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762968/real 1789762968] req@ffff9018d0f26a00 x1876699504279808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762968.254926:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504279808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762968.254931:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762968.254933:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762968.254936:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504279808:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762968.894273:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762969.918202:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762970.942171:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762971.966202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762972.990279:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762973.374183:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762973.374196:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762973.374202:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762973.374214:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2899 00000100:00080000:3.0:1789762973.374228:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762973.374237:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762973.374311:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504279936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762973.374361:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762973.374369:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762973.374373:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762973.374381:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2899 00000100:00080000:3.0:1789762973.374392:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762973.374403:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762973.374421:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762973.374430:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504279936:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789762973.374451:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504280064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762973.374470:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762973.374475:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762973.374477:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762973.374483:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2899 00000100:00080000:3.0:1789762973.374490:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762973.374495:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762973.374502:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762973.374520:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762973.374523:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504280192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762973.374540:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762973.374545:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762973.374548:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762973.374553:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2899 00000100:00080000:3.0:1789762973.374557:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762973.374562:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762973.374571:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789762973.374583:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762973/real 1789762973] req@ffff9018d070ce00 x1876699504279936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762973.374591:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504280320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762973.374658:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504279936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762973.374667:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762973.374674:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762973.374677:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762973.374683:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2899 00000100:00080000:3.0:1789762973.374687:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762973.374697:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762973.374708:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762973.374723:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789762973.374729:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504280448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762973.374740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789762973.374751:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762973.374761:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504279936:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762973.374761:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762973.374766:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762973.374779:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2899 00000100:00080000:3.0:1789762973.374784:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762973.374789:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762973.374794:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762973.374807:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504280576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762973.374825:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504280064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762973.374876:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762973.374906:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762973/real 1789762973] req@ffff9018d070df80 x1876699504280064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762973.374933:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504280064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762973.374952:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762973.374959:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762973.374966:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504280064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762973.374975:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504280192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762973.375010:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762973.375033:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762973/real 1789762973] req@ffff9018d070d880 x1876699504280192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762973.375116:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504280192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762973.375127:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762973.375132:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762973.375136:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504280192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762973.375145:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504280320:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762973.375175:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762973.375190:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762973/real 1789762973] req@ffff9018d070ed80 x1876699504280320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762973.375205:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504280320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762973.375215:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762973.375218:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762973.375222:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504280320:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762973.375229:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504280448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762973.375248:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762973.375259:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762973/real 1789762973] req@ffff9018d070f480 x1876699504280448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762973.375273:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504280448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762973.375313:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762973.375320:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762973.375327:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504280448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762973.375365:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504280576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762973.375405:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762973.375422:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762973/real 1789762973] req@ffff9018d070fb80 x1876699504280576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762973.375437:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504280576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762973.375447:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762973.375452:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762973.375455:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504280576:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762974.014246:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762975.038242:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762976.062234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762977.086264:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762978.110179:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762978.494180:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762978.494191:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762978.494196:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762978.494206:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2905 00000100:00080000:2.0:1789762978.494219:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762978.494227:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762978.494299:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504280704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762978.494326:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762978.494329:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762978.494332:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762978.494335:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2905 00000100:00080000:2.0:1789762978.494339:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762978.494343:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762978.494366:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762978.494376:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504280704:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789762978.494378:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504280832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762978.494391:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762978.494393:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762978.494394:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762978.494397:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2905 00000100:00080000:2.0:1789762978.494399:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762978.494401:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762978.494404:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762978.494412:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504280960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762978.494429:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762978.494432:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762978.494433:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762978.494436:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2905 00000100:00080000:2.0:1789762978.494438:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762978.494441:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762978.494444:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789762978.494452:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504281088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762978.494460:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762978.494463:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762978.494464:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762978.494466:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2905 00000100:00080000:2.0:1789762978.494468:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762978.494471:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762978.494474:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762978.494477:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762978.494496:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504281216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762978.494503:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762978.494506:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789762978.494508:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762978.494512:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2905 00000100:00080000:2.0:1789762978.494515:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762978.494517:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789762978.494520:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762978.494529:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504281344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762978.494541:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762978/real 1789762978] req@ffff9018d0f25c00 x1876699504280704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762978.494569:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504280704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762978.494579:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762978.494582:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762978.494586:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504280704:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762978.494610:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504280832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762978.494638:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762978.494647:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762978/real 1789762978] req@ffff9018d0f27800 x1876699504280832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762978.494655:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504280832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762978.494661:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762978.494664:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762978.494666:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504280832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762978.494669:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504280960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762978.494680:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762978.494685:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762978/real 1789762978] req@ffff9018d0f25180 x1876699504280960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762978.494693:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504280960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762978.494698:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762978.494700:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762978.494701:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504280960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762978.494705:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504281088:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762978.494725:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762978.494732:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762978/real 1789762978] req@ffff9018d0f24380 x1876699504281088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762978.494740:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504281088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762978.494746:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762978.494748:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762978.494750:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504281088:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762978.494753:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504281216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762978.494773:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762978.494778:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762978/real 1789762978] req@ffff9018d0f24e00 x1876699504281216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762978.494786:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504281216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762978.494791:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762978.494793:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762978.494794:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504281216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762978.494798:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504281344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762978.494807:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762978.494813:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762978/real 1789762978] req@ffff9018d0f24000 x1876699504281344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762978.494819:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504281344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762978.494836:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762978.494838:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762978.494840:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504281344:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762979.134190:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762980.158299:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762981.182306:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762982.206210:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762983.230152:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762983.614213:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762983.614227:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762983.614232:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762983.614255:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2910 00000100:00080000:3.0:1789762983.614268:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762983.614275:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762983.614330:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504281472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762983.614358:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762983.614361:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762983.614362:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762983.614366:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2910 00000100:00080000:3.0:1789762983.614368:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762983.614371:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762983.614381:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762983.614391:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504281600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762983.614399:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762983.614402:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762983.614404:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762983.614406:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2910 00000100:00080000:3.0:1789762983.614408:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762983.614412:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762983.614415:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762983.614426:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504281728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762983.614435:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504281472:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762983.614437:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762983.614463:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762983.614465:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762983.614469:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2910 00000100:00080000:3.0:1789762983.614473:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762983.614478:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762983.614485:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762983.614497:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504281856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762983.614508:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762983.614511:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762983.614513:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762983.614517:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2910 00000100:00080000:3.0:1789762983.614523:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762983.614528:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762983.614533:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762983.614534:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762983.614553:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504281984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762983.614563:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762983.614567:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762983.614569:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762983.614574:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2910 00000100:00080000:3.0:1789762983.614578:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762983.614581:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762983.614601:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762983.614606:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762983/real 1789762983] req@ffff9018d070ca80 x1876699504281472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762983.614624:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504282112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762983.614625:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504281472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762983.614698:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762983.614705:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762983.614711:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504281472:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762983.614739:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504281600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762983.614775:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762983.614805:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762983/real 1789762983] req@ffff9018d070d180 x1876699504281600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762983.614813:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504281600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762983.614819:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762983.614821:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762983.614823:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504281600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762983.614844:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504281728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762983.614857:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762983.614863:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762983/real 1789762983] req@ffff9018d070e680 x1876699504281728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762983.614870:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504281728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762983.614875:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762983.614877:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762983.614879:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504281728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762983.614883:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504281856:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762983.614894:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762983.614900:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762983/real 1789762983] req@ffff9018d070f100 x1876699504281856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762983.614907:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504281856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762983.614912:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762983.614914:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762983.614916:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504281856:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762983.614919:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504281984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762983.614930:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762983.614936:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762983/real 1789762983] req@ffff9018d070f800 x1876699504281984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762983.614944:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504281984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762983.614949:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762983.614951:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762983.614953:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504281984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762983.614956:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504282112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762983.614966:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762983.614972:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762983/real 1789762983] req@ffff9018d070d500 x1876699504282112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762983.614978:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504282112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762983.614994:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762983.614996:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762983.614999:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504282112:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762984.254345:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762985.278311:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762986.302264:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762987.326259:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762988.350269:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762988.734120:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762988.734130:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762988.734135:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762988.734146:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2915 00000100:00080000:3.0:1789762988.734181:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762988.734190:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789762988.734233:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504282240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762988.734257:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762988.734260:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762988.734261:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762988.734265:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2915 00000100:00080000:3.0:1789762988.734268:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762988.734270:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762988.734280:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762988.734290:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504282368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762988.734298:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762988.734302:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789762988.734305:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504282240:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789762988.734305:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762988.734312:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2915 00000100:00080000:3.0:1789762988.734316:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762988.734318:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762988.734322:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762988.734329:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504282496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762988.734335:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762988.734337:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762988.734339:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762988.734342:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2915 00000100:00080000:3.0:1789762988.734345:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762988.734347:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762988.734350:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789762988.734359:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504282624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762988.734365:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762988.734368:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762988.734370:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762988.734372:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2915 00000100:00080000:3.0:1789762988.734375:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762988.734377:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762988.734381:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762988.734384:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762988.734391:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504282752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762988.734403:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762988.734406:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789762988.734408:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762988.734410:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2915 00000100:00080000:3.0:1789762988.734412:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762988.734415:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789762988.734418:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762988.734426:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504282880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762988.734433:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762988/real 1789762988] req@ffff9018d070e300 x1876699504282240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762988.734450:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504282240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762988.734464:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762988.734468:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762988.734472:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504282240:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762988.734493:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504282368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762988.734508:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762988.734517:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762988/real 1789762988] req@ffff9018d070c000 x1876699504282368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762988.734525:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504282368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762988.734530:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762988.734532:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762988.734534:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504282368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762988.734538:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504282496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762988.734549:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762988.734554:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762988/real 1789762988] req@ffff9018d070dc00 x1876699504282496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762988.734562:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504282496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762988.734567:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762988.734569:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762988.734571:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504282496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762988.734575:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504282624:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762988.734587:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762988.734593:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762988/real 1789762988] req@ffff9018d070d500 x1876699504282624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762988.734600:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504282624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762988.734606:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762988.734607:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762988.734609:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504282624:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762988.734612:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504282752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762988.734622:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762988.734628:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762988/real 1789762988] req@ffff9018d070f800 x1876699504282752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762988.734634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504282752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762988.734639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762988.734641:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762988.734643:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504282752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762988.734646:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504282880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762988.734656:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762988.734662:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762988/real 1789762988] req@ffff9018d070f100 x1876699504282880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762988.734669:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504282880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762988.734674:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762988.734676:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762988.734678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504282880:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762989.374226:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762990.398207:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762991.422263:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762992.446258:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762993.470260:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762993.854250:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762993.854261:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762993.854266:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762993.854279:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2920 00000100:00080000:0.0:1789762993.854291:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762993.854298:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762993.854344:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504283008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762993.854369:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762993.854372:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762993.854373:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762993.854376:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2920 00000100:00080000:0.0:1789762993.854379:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762993.854381:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762993.854390:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762993.854400:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504283136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762993.854408:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762993.854411:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762993.854412:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762993.854415:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2920 00000100:00080000:0.0:1789762993.854419:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762993.854422:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762993.854426:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762993.854432:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504283008:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789762993.854438:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504283264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762993.854446:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762993.854448:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762993.854450:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762993.854452:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2920 00000100:00080000:0.0:1789762993.854454:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762993.854457:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762993.854461:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762993.854469:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504283392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762993.854476:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762993.854478:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762993.854479:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762993.854481:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2920 00000100:00080000:0.0:1789762993.854484:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762993.854487:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762993.854490:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762993.854497:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504283520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762993.854508:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762993.854511:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762993.854512:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762993.854515:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2920 00000100:00080000:0.0:1789762993.854517:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762993.854520:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789762993.854525:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789762993.854529:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762993.854563:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504283648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762993.854613:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762993/real 1789762993] req@ffff9018c8b9b100 x1876699504283008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762993.854657:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504283008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762993.854674:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762993.854680:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762993.854686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504283008:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789762993.854696:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504283136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762993.854737:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762993.854747:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762993/real 1789762993] req@ffff9018c8b9b480 x1876699504283136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762993.854755:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504283136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762993.854766:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762993.854768:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762993.854771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504283136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762993.854819:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504283264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762993.854842:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762993.854854:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762993/real 1789762993] req@ffff9018c8b98000 x1876699504283264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762993.854868:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504283264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762993.854879:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762993.854882:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762993.854887:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504283264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762993.854894:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504283392:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789762993.854927:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762993.854933:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762993/real 1789762993] req@ffff9018c8b9aa00 x1876699504283392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762993.854940:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504283392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762993.854946:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762993.854948:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762993.854951:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504283392:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789762993.854954:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504283520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762993.854965:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762993.854971:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762993/real 1789762993] req@ffff9018c8b98e00 x1876699504283520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762993.854977:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504283520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762993.854982:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762993.854983:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762993.854985:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504283520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789762993.854989:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504283648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789762993.855016:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789762993.855028:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762993/real 1789762993] req@ffff9018c8b9a680 x1876699504283648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762993.855042:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504283648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762993.855052:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762993.855055:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789762993.855058:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504283648:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762994.494256:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762995.518281:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762996.542262:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762997.566255:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762998.590195:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762998.974234:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762998.974254:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762998.974269:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762998.974290:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2925 00000100:00080000:0.0:1789762998.974308:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762998.974314:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789762998.974348:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504283776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762998.974394:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762998.974397:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762998.974398:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762998.974402:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2925 00000100:00080000:0.0:1789762998.974405:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762998.974416:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504283776:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789762998.974422:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762998.974441:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762998.974471:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504283904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762998.974491:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762998.974495:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1789762998.974501:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762998.974502:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762998.974528:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2925 00000100:00080000:0.0:1789762998.974534:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762998.974546:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762998.974547:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762998/real 1789762998] req@ffff9018c8b9a300 x1876699504283776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789762998.974554:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762998.974571:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504284032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762998.974581:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762998.974583:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504283776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762998.974584:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762998.974592:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762998.974612:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2925 00000100:00080000:0.0:1789762998.974620:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762998.974623:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762998.974630:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789762998.974633:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762998.974636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504283776:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789762998.974640:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789762998.974668:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504284160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762998.974673:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504283904:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789762998.974698:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762998.974702:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762998.974704:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762998.974712:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762998.974721:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2925 00000100:00100000:2.0:1789762998.974728:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762998/real 1789762998] req@ffff9018c8b9ad80 x1876699504283904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762998.974748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504283904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762998.974750:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762998.974762:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762998.974770:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789762998.974776:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789762998.974778:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504283904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789762998.974793:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504284032:192.168.203.158@tcp:38:kworker.0 00000008:00100000:0.0:1789762998.974815:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762998.974822:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762998.974832:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762998/real 1789762998] req@ffff9018c8b99500 x1876699504284032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762998.974842:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504284032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762998.974850:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762998.974853:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762998.974856:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504284032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789762998.974871:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504284288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762998.974877:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504284160:192.168.203.158@tcp:8:kworker.0 00000100:00080000:0.0:1789762998.974906:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762998.974909:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789762998.974924:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789762998.974927:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762998.974934:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2925 00000100:00100000:2.0:1789762998.974934:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762998/real 1789762998] req@ffff9018c8b99c00 x1876699504284160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762998.974950:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762998.974955:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504284160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762998.974960:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789762998.974971:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762998.974975:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762998.974979:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789762998.974982:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504284160:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789762998.975005:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504284288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789762998.975024:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789762998.975061:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762998/real 1789762998] req@ffff9018c8b98380 x1876699504284288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762998.975074:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504284288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762998.975081:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762998.975085:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789762998.975086:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504284416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762998.975103:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504284288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789762998.975176:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504284416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789762998.975215:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789762998.975239:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762998/real 1789762998] req@ffff9018c8b99500 x1876699504284416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762998.975257:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504284416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762998.975267:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762998.975271:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789762998.975275:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504284416:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789762999.614204:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763000.638167:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763001.662179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763002.686168:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763003.710210:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763004.094119:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763004.094130:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763004.094134:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763004.094146:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2930 00000100:00080000:2.0:1789763004.094157:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763004.094164:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763004.094206:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504284544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763004.094229:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763004.094232:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763004.094234:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763004.094237:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2930 00000100:00080000:2.0:1789763004.094240:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763004.094243:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763004.094252:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763004.094262:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504284672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763004.094271:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504284544:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763004.094272:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763004.094280:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763004.094282:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763004.094285:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2930 00000100:00080000:2.0:1789763004.094288:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763004.094290:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763004.094293:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763004.094302:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504284800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763004.094308:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763004.094310:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763004.094312:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763004.094315:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2930 00000100:00080000:2.0:1789763004.094336:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763004.094339:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763004.094344:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789763004.094348:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763004.094356:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504284928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763004.094364:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763004.094366:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763004.094368:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763004.094370:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2930 00000100:00080000:2.0:1789763004.094373:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763004.094376:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763004.094378:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763004.094387:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504285056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763004.094394:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763004.094397:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763004.094398:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763004.094401:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2930 00000100:00100000:3.0:1789763004.094405:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763004/real 1789763004] req@ffff9018d0f25500 x1876699504284544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763004.094406:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763004.094412:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763004.094418:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763004.094429:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504285184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763004.094434:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504284544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763004.094454:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763004.094457:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763004.094461:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504284544:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763004.094484:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504284672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763004.094507:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763004.094522:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763004/real 1789763004] req@ffff9018d0f27480 x1876699504284672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763004.094536:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504284672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763004.094544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763004.094547:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763004.094549:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504284672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763004.094554:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504284800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763004.094577:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763004.094586:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763004/real 1789763004] req@ffff9018d0f26680 x1876699504284800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763004.094594:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504284800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763004.094605:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763004.094607:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763004.094609:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504284800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763004.094613:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504284928:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763004.094625:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763004.094630:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763004/real 1789763004] req@ffff9018d0f24700 x1876699504284928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763004.094637:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504284928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763004.094643:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763004.094645:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763004.094647:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504284928:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763004.094650:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504285056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763004.094661:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763004.094667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763004/real 1789763004] req@ffff9018d0f24000 x1876699504285056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763004.094674:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504285056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763004.094679:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763004.094681:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763004.094683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504285056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763004.094687:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504285184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763004.094697:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763004.094702:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763004/real 1789763004] req@ffff9018d0f24e00 x1876699504285184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763004.094710:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504285184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763004.094715:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763004.094717:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763004.094718:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504285184:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763004.734226:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763005.758169:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763006.782163:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763007.806201:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763008.830221:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763009.214150:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763009.214161:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763009.214166:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763009.214180:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2935 00000100:00080000:0.0:1789763009.214193:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763009.214201:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763009.214244:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504285312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763009.214282:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763009.214290:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763009.214291:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763009.214296:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2935 00000100:00080000:0.0:1789763009.214300:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763009.214305:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763009.214316:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763009.214327:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504285312:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763009.214332:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504285440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763009.214345:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763009.214349:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763009.214351:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763009.214356:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2935 00000100:00080000:0.0:1789763009.214359:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763009.214363:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763009.214368:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763009.214381:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504285568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763009.214391:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763009.214395:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763009.214398:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789763009.214399:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763009.214407:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2935 00000100:00080000:0.0:1789763009.214414:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763009.214419:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763009.214425:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763009.214436:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504285696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763009.214448:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763009.214454:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763009/real 1789763009] req@ffff9018c8b9ad80 x1876699504285312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763009.214455:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763009.214459:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763009.214468:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2935 00000100:00080000:0.0:1789763009.214474:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763009.214479:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763009.214483:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504285312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763009.214486:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763009.214502:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504285824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763009.214504:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763009.214512:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763009.214520:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504285312:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763009.214522:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763009.214530:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763009.214533:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763009.214537:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2935 00000100:00080000:0.0:1789763009.214540:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763009.214545:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763009.214550:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763009.214559:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504285440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763009.214563:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504285952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763009.214596:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763009.214611:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763009/real 1789763009] req@ffff9018c8b9a300 x1876699504285440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763009.214621:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504285440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763009.214628:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763009.214630:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763009.214633:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504285440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763009.214637:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504285568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763009.214649:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763009.214655:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763009/real 1789763009] req@ffff9018c8b9a680 x1876699504285568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763009.214663:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504285568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763009.214668:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763009.214670:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763009.214672:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504285568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763009.214676:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504285696:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763009.214687:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763009.214693:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763009/real 1789763009] req@ffff9018c8b98e00 x1876699504285696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763009.214701:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504285696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763009.214707:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763009.214709:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763009.214711:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504285696:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763009.214714:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504285824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763009.214724:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763009.214730:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763009/real 1789763009] req@ffff9018c8b9aa00 x1876699504285824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763009.214736:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504285824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763009.214741:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763009.214743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763009.214744:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504285824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763009.214756:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504285952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763009.214767:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763009.214773:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763009/real 1789763009] req@ffff9018c8b98000 x1876699504285952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763009.214781:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504285952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763009.214786:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763009.214788:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763009.214791:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504285952:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763009.854158:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763010.878166:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763011.902200:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763012.926183:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763013.950276:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763014.334117:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763014.334125:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763014.334127:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763014.334135:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2940 00000100:00080000:3.0:1789763014.334144:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763014.334149:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763014.334180:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504286080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763014.334204:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763014.334207:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763014.334208:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763014.334211:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2940 00000100:00080000:3.0:1789763014.334214:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763014.334216:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763014.334224:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763014.334231:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504286208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763014.334238:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763014.334241:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763014.334242:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763014.334245:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2940 00000100:00080000:3.0:1789763014.334247:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763014.334250:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763014.334252:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763014.334260:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504286336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763014.334266:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763014.334269:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763014.334270:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763014.334272:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2940 00000100:00080000:3.0:1789763014.334273:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763014.334276:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763014.334279:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763014.334286:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504286464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763014.334293:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763014.334296:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763014.334302:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763014.334303:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504286080:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763014.334307:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2940 00000100:00080000:3.0:1789763014.334320:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763014.334324:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763014.334327:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763014.334336:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504286592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763014.334342:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763014.334346:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763014.334347:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763014.334349:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2940 00000100:00080000:3.0:1789763014.334354:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763014.334356:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763014.334359:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763014.334365:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504286720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763014.334395:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763014.334440:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763014/real 1789763014] req@ffff9018d070e680 x1876699504286080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763014.334456:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504286080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763014.334465:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763014.334469:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763014.334473:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504286080:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763014.334478:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504286208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763014.334493:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763014.334501:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763014/real 1789763014] req@ffff9018d070d180 x1876699504286208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763014.334510:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504286208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763014.334516:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763014.334519:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763014.334521:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504286208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763014.334524:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504286336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763014.334535:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763014.334541:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763014/real 1789763014] req@ffff9018d070ca80 x1876699504286336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763014.334548:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504286336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763014.334553:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763014.334555:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763014.334557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504286336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763014.334589:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504286464:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763014.334609:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763014.334621:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763014/real 1789763014] req@ffff9018d070fb80 x1876699504286464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763014.334633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504286464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763014.334642:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763014.334646:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763014.334649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504286464:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763014.334655:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504286592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763014.334671:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763014.334679:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763014/real 1789763014] req@ffff9018d070f480 x1876699504286592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763014.334687:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504286592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763014.334692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763014.334694:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763014.334697:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504286592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763014.334700:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504286720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763014.334710:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763014.334716:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763014/real 1789763014] req@ffff9018d070ed80 x1876699504286720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763014.334723:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504286720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763014.334728:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763014.334730:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763014.334731:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504286720:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763014.974209:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763015.998218:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763017.022264:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763018.046196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763019.070339:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763019.454190:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763019.454206:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763019.454213:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763019.454229:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2946 00000100:00080000:2.0:1789763019.454247:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763019.454258:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763019.454322:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504286848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763019.454368:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763019.454374:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763019.454377:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763019.454383:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2946 00000100:00080000:2.0:1789763019.454387:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763019.454394:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763019.454425:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763019.454441:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504286976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763019.454459:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763019.454461:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504286848:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763019.454467:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763019.454474:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763019.454483:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2946 00000100:00080000:2.0:1789763019.454489:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763019.454497:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763019.454505:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763019.454525:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504287104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763019.454538:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763019.454542:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763019.454544:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763019.454548:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2946 00000100:00080000:2.0:1789763019.454555:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763019.454559:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763019.454565:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789763019.454572:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763019.454580:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504287232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763019.454593:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763019.454596:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763019.454598:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763019.454602:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2946 00000100:00080000:2.0:1789763019.454604:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763019.454608:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763019.454613:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763019.454625:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504287360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763019.454634:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763019/real 1789763019] req@ffff9018d0f24380 x1876699504286848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763019.454636:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763019.454647:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763019.454652:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763019.454661:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2946 00000100:00080000:2.0:1789763019.454664:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763019.454667:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763019.454674:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763019.454685:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504287488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763019.454692:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504286848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763019.454710:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763019.454715:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763019.454721:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504286848:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763019.454754:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504286976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763019.454779:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763019.454792:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763019/real 1789763019] req@ffff9018d0f25180 x1876699504286976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763019.454825:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504286976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763019.454838:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763019.454843:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763019.454847:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504286976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763019.454854:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504287104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763019.454877:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763019.454888:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763019/real 1789763019] req@ffff9018d0f27800 x1876699504287104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763019.454903:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504287104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763019.454913:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763019.454917:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763019.454922:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504287104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763019.454929:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504287232:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763019.454982:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763019.454998:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763019/real 1789763019] req@ffff9018d0f25c00 x1876699504287232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763019.455014:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504287232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763019.455025:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763019.455029:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763019.455065:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504287232:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763019.455073:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504287360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763019.455100:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763019.455115:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763019/real 1789763019] req@ffff9018d0f26a00 x1876699504287360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763019.455131:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504287360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763019.455141:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763019.455145:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763019.455150:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504287360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763019.455157:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504287488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763019.455193:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763019.455207:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763019/real 1789763019] req@ffff9018d0f24a80 x1876699504287488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763019.455222:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504287488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763019.455232:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763019.455236:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763019.455240:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504287488:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763020.094229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763021.118261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763022.142214:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763023.166257:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763024.190250:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763024.574222:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763024.574238:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763024.574244:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763024.574259:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2951 00000100:00080000:0.0:1789763024.574276:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763024.574287:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763024.574350:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504287616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763024.574390:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763024.574396:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763024.574399:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763024.574404:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2951 00000100:00080000:0.0:1789763024.574408:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763024.574413:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763024.574427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763024.574443:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504287744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763024.574457:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763024.574462:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763024.574464:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763024.574471:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2951 00000100:00080000:0.0:1789763024.574474:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763024.574479:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763024.574487:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763024.574498:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504287872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763024.574512:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763024.574516:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763024.574521:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763024.574531:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504287616:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763024.574532:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2951 00000100:00080000:0.0:1789763024.574536:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763024.574553:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763024.574566:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763024.574580:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504288000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763024.574593:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763024.574598:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763024.574601:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763024.574606:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2951 00000100:00080000:0.0:1789763024.574615:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763024.574622:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763024.574627:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763024.574646:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504288128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763024.574658:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763024.574663:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763024.574666:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763024.574671:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2951 00000100:00080000:0.0:1789763024.574676:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763024.574682:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763024.574690:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763024.574707:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504287744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763024.574710:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504288256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763024.574805:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504287872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763024.574851:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504288000:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763024.574875:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504288128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763024.574897:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504288256:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763025.214273:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763026.238239:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763027.262286:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763027.646476:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789763027.646538:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763027.646566:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763027.646625:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763027.646645:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763027.646663:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763027.646680:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763027.646695:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789763027.646700:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763024/real 1789763027] req@ffff9018c8b9b800 x1876699504287872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763027.646750:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504287872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763027.646771:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763027.646779:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763027.646786:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504287872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763027.646800:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763024/real 1789763027] req@ffff9018c8b99880 x1876699504288000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763027.646824:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504288000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763027.646837:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763027.646843:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763027.646858:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504288000:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763027.646869:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763024/real 1789763027] req@ffff9018c8b98a80 x1876699504288128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763027.646925:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504288128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763027.646940:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763027.646947:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763027.646954:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504288128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763027.646965:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763024/real 1789763027] req@ffff9018c8b99f80 x1876699504288256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763027.646985:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504288256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763027.646996:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763027.647001:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763027.647006:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504288256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763027.647015:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763024/real 1789763027] req@ffff9018c8b9b480 x1876699504287616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763027.647072:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504287616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763027.647089:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763027.647095:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763027.647102:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504287616:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763027.647118:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763024/real 1789763027] req@ffff9018c8b9b100 x1876699504287744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763027.647141:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504287744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763027.647155:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763027.647160:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763027.647165:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504287744:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763028.286248:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763029.310303:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763029.694239:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763029.694253:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763029.694259:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763029.694273:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2956 00000100:00080000:0.0:1789763029.694288:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763029.694298:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763029.694353:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504288384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763029.694386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763029.694391:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763029.694394:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763029.694399:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2956 00000100:00080000:0.0:1789763029.694403:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763029.694408:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763029.694423:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763029.694447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504288512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763029.694484:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763029.694490:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504288384:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763029.694494:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763029.694504:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763029.694517:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2956 00000100:00080000:0.0:1789763029.694527:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763029.694535:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763029.694544:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763029.694565:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504288640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763029.694579:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763029.694584:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763029.694587:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789763029.694598:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763029.694602:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2956 00000100:00080000:0.0:1789763029.694611:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763029.694616:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763029.694624:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763029.694639:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504288768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763029.694653:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763029.694658:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763029.694660:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763029.694667:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2956 00000100:00080000:0.0:1789763029.694674:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763029.694684:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763029.694686:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763029/real 1789763029] req@ffff9018c8b98700 x1876699504288384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763029.694690:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763029.694710:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504288896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763029.694728:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763029.694733:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763029.694736:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763029.694744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504288384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763029.694748:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2956 00000100:00080000:0.0:1789763029.694756:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763029.694766:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763029.694778:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789763029.694793:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789763029.694797:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504289024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763029.694808:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763029.694825:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504288384:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763029.694915:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504288512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763029.694967:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763029.695010:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763029/real 1789763029] req@ffff9018c8b9bb80 x1876699504288512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763029.695118:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504288512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763029.695150:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763029.695157:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763029.695165:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504288512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763029.695178:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504288640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763029.695226:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763029.695252:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763029/real 1789763029] req@ffff9018c8b9b100 x1876699504288640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763029.695278:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504288640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763029.695294:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763029.695300:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763029.695307:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504288640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763029.695319:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504288768:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763029.695361:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763029.695384:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763029/real 1789763029] req@ffff9018c8b9b480 x1876699504288768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763029.695410:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504288768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763029.695426:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763029.695432:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763029.695439:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504288768:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763029.695451:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504288896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763029.695778:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763029.695807:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763029/real 1789763029] req@ffff9018c8b99f80 x1876699504288896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763029.695833:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504288896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763029.695850:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763029.695856:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763029.695888:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504288896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763029.695905:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504289024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763029.695950:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763029.695976:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763029/real 1789763029] req@ffff9018c8b98a80 x1876699504289024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763029.696001:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504289024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763029.696017:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763029.696024:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763029.696031:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504289024:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763030.334247:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763031.358294:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763032.382252:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763033.406276:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763034.430280:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763034.814253:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763034.814268:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763034.814277:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763034.814300:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2961 00000100:00080000:1.0:1789763034.814319:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763034.814331:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763034.814389:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504289152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763034.814432:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763034.814440:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763034.814444:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763034.814454:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2961 00000100:00080000:1.0:1789763034.814464:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763034.814474:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763034.814495:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763034.814513:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504289152:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763034.814533:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504289280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763034.814558:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763034.814564:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763034.814566:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763034.814573:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2961 00000100:00080000:1.0:1789763034.814582:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763034.814590:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763034.814598:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763034.814618:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504289408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763034.814632:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763034.814663:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763034.814668:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763034.814672:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763034.814678:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2961 00000100:00080000:1.0:1789763034.814686:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763034.814693:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763034.814703:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763034.814722:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504289536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763034.814736:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763034.814742:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763034.814745:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763034.814752:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2961 00000100:00080000:1.0:1789763034.814759:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763034.814766:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763034.814774:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763034.814793:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504289664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763034.814813:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763034.814820:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763034.814824:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763034.814852:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2961 00000100:00080000:1.0:1789763034.814859:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763034.814871:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763034/real 1789763034] req@ffff9018c66a7800 x1876699504289152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763034.814872:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763034.814884:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763034.814918:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504289792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763034.814936:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504289152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763034.814956:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763034.814965:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763034.814981:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504289152:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763034.815278:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504289280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763034.815326:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763034.815435:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763034/real 1789763034] req@ffff9018c66a7b80 x1876699504289280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763034.815460:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504289280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763034.815476:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763034.815485:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763034.815492:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504289280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763034.815507:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504289408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763034.815552:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763034.815578:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763034/real 1789763034] req@ffff9018c66a4000 x1876699504289408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763034.815603:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504289408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763034.815634:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763034.815641:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763034.815648:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504289408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763034.815659:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504289536:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763034.815693:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763034.815709:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763034/real 1789763034] req@ffff9018c66a5f80 x1876699504289536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763034.815725:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504289536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763034.815735:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763034.815739:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763034.815743:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504289536:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763034.815752:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504289664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763034.815785:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763034.815806:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763034/real 1789763034] req@ffff9018c66a6d80 x1876699504289664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763034.815861:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504289664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763034.815880:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763034.815886:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763034.815891:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504289664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763034.815901:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504289792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763034.815932:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763034.815948:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763034/real 1789763034] req@ffff9018c66a5c00 x1876699504289792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763034.815968:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504289792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763034.815984:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763034.815991:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763034.815997:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504289792:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763035.454249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763036.478159:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763037.502161:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763038.526262:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763039.550250:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763039.934297:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763039.934318:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763039.934324:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763039.934341:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2966 00000100:00080000:3.0:1789763039.934358:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763039.934370:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763039.934435:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504289920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763039.934497:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763039.934503:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763039.934505:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763039.934511:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2966 00000100:00080000:3.0:1789763039.934515:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763039.934521:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763039.934536:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763039.934553:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504290048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763039.934567:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763039.934575:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789763039.934578:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504289920:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763039.934579:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763039.934592:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2966 00000100:00080000:3.0:1789763039.934599:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763039.934603:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763039.934608:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763039.934621:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504290176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763039.934633:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763039.934637:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763039.934639:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763039.934643:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2966 00000100:00080000:3.0:1789763039.934647:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763039.934652:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763039.934657:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763039.934672:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504290304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763039.934683:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763039.934688:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763039.934691:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763039.934696:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2966 00000100:00080000:3.0:1789763039.934700:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763039.934704:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763039.934709:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763039.934719:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763039.934725:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504290432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763039.934739:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763039.934743:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763039.934746:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763039.934753:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2966 00000100:00080000:3.0:1789763039.934760:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763039.934764:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763039.934769:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763039.934782:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504290560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763039.934823:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763039/real 1789763039] req@ffff9018d070d880 x1876699504289920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763039.934865:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504289920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763039.934882:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763039.934887:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763039.934893:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504289920:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763039.934931:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504290048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763039.934966:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763039.934983:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763039/real 1789763039] req@ffff9018d070df80 x1876699504290048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763039.934998:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504290048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763039.935008:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763039.935014:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763039.935018:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504290048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763039.935025:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504290176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763039.935084:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763039.935102:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763039/real 1789763039] req@ffff9018d070ce00 x1876699504290176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763039.935125:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504290176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763039.935140:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763039.935146:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763039.935153:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504290176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763039.935165:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504290304:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763039.935204:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763039.935219:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763039/real 1789763039] req@ffff9018d070ed80 x1876699504290304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763039.935233:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504290304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763039.935244:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763039.935248:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763039.935252:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504290304:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763039.935259:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504290432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763039.935297:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763039.935309:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763039/real 1789763039] req@ffff9018d070f480 x1876699504290432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763039.935323:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504290432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763039.935332:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763039.935336:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763039.935339:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504290432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763039.935346:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504290560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763039.935366:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763039.935382:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763039/real 1789763039] req@ffff9018d070fb80 x1876699504290560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763039.935424:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504290560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763039.935440:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763039.935447:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763039.935454:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504290560:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763040.574270:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763041.598283:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763042.622220:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763043.646224:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763044.670250:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763045.054148:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763045.054163:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763045.054168:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763045.054182:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2971 00000100:00080000:2.0:1789763045.054197:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763045.054205:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763045.054259:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504290688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763045.054286:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763045.054290:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763045.054291:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763045.054295:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2971 00000100:00080000:2.0:1789763045.054299:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763045.054303:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763045.054315:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763045.054323:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504290688:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763045.054334:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504290816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763045.054348:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763045.054352:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763045.054354:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763045.054359:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2971 00000100:00080000:2.0:1789763045.054364:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763045.054370:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763045.054374:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763045.054385:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504290944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763045.054396:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763045.054402:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763045.054406:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763045.054407:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763045.054411:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2971 00000100:00080000:2.0:1789763045.054414:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763045.054416:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763045.054420:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763045.054430:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504291072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763045.054437:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763045.054440:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763045.054441:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763045.054443:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2971 00000100:00080000:2.0:1789763045.054445:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763045.054448:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763045.054451:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763045.054461:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504291200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763045.054465:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763045/real 1789763045] req@ffff9018d0f26300 x1876699504290688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763045.054471:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763045.054478:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763045.054481:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763045.054487:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2971 00000100:00080000:2.0:1789763045.054491:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763045.054494:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763045.054497:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763045.054498:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504290688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763045.054507:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504291328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763045.054529:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763045.054535:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763045.054539:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504290688:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763045.054562:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504290816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763045.054581:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763045.054591:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763045/real 1789763045] req@ffff9018d0f27100 x1876699504290816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763045.054600:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504290816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763045.054606:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763045.054608:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763045.054610:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504290816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763045.054613:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504290944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763045.054624:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763045.054630:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763045/real 1789763045] req@ffff9018d0f25f80 x1876699504290944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763045.054638:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504290944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763045.054643:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763045.054645:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763045.054646:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504290944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763045.054651:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504291072:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763045.054662:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763045.054668:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763045/real 1789763045] req@ffff9018d0f26d80 x1876699504291072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763045.054675:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504291072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763045.054681:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763045.054682:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763045.054684:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504291072:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763045.054688:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504291200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763045.054698:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763045.054704:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763045/real 1789763045] req@ffff9018d0f24a80 x1876699504291200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763045.054711:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504291200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763045.054717:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763045.054719:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763045.054720:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504291200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763045.054723:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504291328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763045.054733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763045.054738:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763045/real 1789763045] req@ffff9018d0f26a00 x1876699504291328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763045.054746:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504291328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763045.054772:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763045.054776:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763045.054779:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504291328:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763045.694248:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763046.718180:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763047.742289:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763048.766267:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763049.790234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763050.174225:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763050.174243:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763050.174249:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763050.174267:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2976 00000100:00080000:0.0:1789763050.174285:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763050.174297:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763050.174349:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504291456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763050.174391:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763050.174397:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763050.174400:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763050.174405:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2976 00000100:00080000:0.0:1789763050.174410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763050.174414:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763050.174428:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763050.174446:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504291584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763050.174460:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763050.174466:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763050.174487:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763050.174492:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2976 00000100:00080000:0.0:1789763050.174498:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763050.174503:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763050.174509:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763050.174523:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504291712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763050.174537:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504291456:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763050.174542:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763050.174571:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763050.174576:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763050.174582:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2976 00000100:00080000:0.0:1789763050.174586:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763050.174610:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763050.174619:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763050.174635:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504291840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763050.174648:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763050.174653:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763050.174655:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763050.174660:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2976 00000100:00080000:0.0:1789763050.174664:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763050.174670:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763050.174677:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763050.174689:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504291968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763050.174707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763050.174762:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763050.174766:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763050.174770:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763050.174779:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2976 00000100:00080000:0.0:1789763050.174787:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763050.174792:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763050.174799:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763050.174817:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504292096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763050.174832:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763050/real 1789763050] req@ffff9018c8b99880 x1876699504291456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763050.174874:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504291456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763050.174893:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763050.174900:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763050.174906:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504291456:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763050.174918:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504291584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763050.174955:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763050.174984:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763050/real 1789763050] req@ffff9018c8b9b800 x1876699504291584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763050.175008:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504291584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763050.175104:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763050.175111:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763050.175116:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504291584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763050.175174:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504291712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763050.175210:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763050.175227:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763050/real 1789763050] req@ffff9018c8b98000 x1876699504291712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763050.175243:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504291712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763050.175254:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763050.175259:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763050.175263:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504291712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763050.175271:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504291840:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763050.175292:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763050.175304:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763050/real 1789763050] req@ffff9018c8b9aa00 x1876699504291840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763050.175318:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504291840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763050.175328:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763050.175332:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763050.175337:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504291840:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763050.175345:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504291968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763050.175380:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763050.175403:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763050/real 1789763050] req@ffff9018c8b98e00 x1876699504291968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763050.175429:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504291968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763050.175446:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763050.175453:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763050.175461:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504291968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763050.175474:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504292096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763050.175519:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763050.175536:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763050/real 1789763050] req@ffff9018c8b9a680 x1876699504292096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763050.175552:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504292096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763050.175562:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763050.175567:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763050.175570:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504292096:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763050.814231:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763051.838274:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763052.862256:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763053.886323:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763054.910290:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763055.294246:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763055.294268:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763055.294276:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763055.294295:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2981 00000100:00080000:0.0:1789763055.294314:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763055.294326:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763055.294393:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504292224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763055.294431:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763055.294436:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763055.294438:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763055.294443:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2981 00000100:00080000:0.0:1789763055.294448:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763055.294454:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763055.294470:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763055.294488:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504292224:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763055.294491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504292352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763055.294513:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763055.294517:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763055.294519:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763055.294524:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2981 00000100:00080000:0.0:1789763055.294528:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763055.294531:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763055.294537:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763055.294551:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504292480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763055.294589:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763055.294594:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763055.294597:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763055.294602:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2981 00000100:00080000:0.0:1789763055.294608:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763055.294612:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763055.294619:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789763055.294628:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763055.294640:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504292608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763055.294655:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763055.294660:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763055.294663:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763055.294669:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2981 00000100:00080000:0.0:1789763055.294673:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763055.294720:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763055.294728:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763055.294754:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504292736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763055.294757:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763055/real 1789763055] req@ffff9018c8b9a300 x1876699504292224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763055.294812:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763055.294819:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763055.294821:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763055.294829:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2981 00000100:00080000:0.0:1789763055.294836:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763055.294839:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504292224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763055.294847:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763055.294865:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789763055.294872:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763055.294885:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763055.294904:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504292864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763055.294920:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504292224:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763055.294975:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504292352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763055.295020:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763055.295039:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763055/real 1789763055] req@ffff9018c8b9ad80 x1876699504292352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763055.295056:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504292352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763055.295106:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763055.295111:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763055.295115:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504292352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763055.295123:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504292480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763055.295148:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763055.295162:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763055/real 1789763055] req@ffff9018c8b99500 x1876699504292480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763055.295195:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504292480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763055.295214:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763055.295218:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763055.295222:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504292480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763055.295230:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504292608:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763055.295270:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763055.295280:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763055/real 1789763055] req@ffff9018c8b98380 x1876699504292608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763055.295293:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504292608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763055.295303:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763055.295306:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763055.295310:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504292608:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763055.295315:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504292736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763055.295333:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763055.295343:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763055/real 1789763055] req@ffff9018c8b99c00 x1876699504292736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763055.295356:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504292736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763055.295365:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763055.295368:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763055.295372:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504292736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763055.295379:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504292864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763055.295396:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763055.295406:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763055/real 1789763055] req@ffff9018c8b9a680 x1876699504292864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763055.295418:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504292864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763055.295427:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763055.295430:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763055.295434:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504292864:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763055.934165:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763056.958244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763057.982237:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763059.006333:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763060.030254:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763060.414276:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763060.414286:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763060.414290:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763060.414299:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2986 00000100:00080000:0.0:1789763060.414309:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763060.414316:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763060.414356:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504292992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763060.414385:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763060.414390:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763060.414393:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763060.414397:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2986 00000100:00080000:0.0:1789763060.414403:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763060.414409:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763060.414419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763060.414429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504293120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763060.414438:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504292992:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763060.414442:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763060.414447:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763060.414448:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763060.414452:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2986 00000100:00080000:0.0:1789763060.414455:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763060.414457:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763060.414461:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763060.414472:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504293248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763060.414480:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763060.414483:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763060.414484:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763060.414487:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2986 00000100:00080000:0.0:1789763060.414491:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763060.414494:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763060.414497:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789763060.414503:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763060.414507:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504293376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763060.414516:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763060.414520:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763060.414522:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763060.414524:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2986 00000100:00080000:0.0:1789763060.414527:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763060.414556:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763060.414563:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763060.414577:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504293504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763060.414589:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763060.414593:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763060/real 1789763060] req@ffff9018c8b98e00 x1876699504292992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763060.414595:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763060.414601:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763060.414615:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2986 00000100:00080000:0.0:1789763060.414624:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763060.414632:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763060.414679:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504292992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763060.414681:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763060.414697:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504293632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763060.414699:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763060.414712:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763060.414720:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504292992:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763060.414772:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504293120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763060.414804:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763060.414819:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763060/real 1789763060] req@ffff9018c8b9aa00 x1876699504293120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763060.414833:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504293120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763060.414842:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763060.414845:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763060.414849:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504293120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763060.414855:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504293248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763060.414878:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763060.414895:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763060/real 1789763060] req@ffff9018c8b98000 x1876699504293248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763060.414914:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504293248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763060.414928:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763060.414934:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763060.414940:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504293248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763060.414950:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504293376:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763060.414976:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763060.414988:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763060/real 1789763060] req@ffff9018c8b9b800 x1876699504293376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763060.415000:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504293376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763060.415008:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763060.415027:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763060.415030:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504293376:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763060.415034:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504293504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763060.415048:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763060.415055:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763060/real 1789763060] req@ffff9018c8b99880 x1876699504293504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763060.415063:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504293504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763060.415069:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763060.415071:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763060.415073:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504293504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763060.415077:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504293632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763060.415125:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763060.415137:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763060/real 1789763060] req@ffff9018c8b98a80 x1876699504293632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763060.415162:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504293632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763060.415192:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763060.415196:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763060.415200:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504293632:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763061.054266:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763062.078243:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763063.102264:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763064.126326:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763065.150325:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763065.534196:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763065.534212:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763065.534219:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763065.534236:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2992 00000100:00080000:0.0:1789763065.534255:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763065.534266:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763065.534334:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504293760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763065.534388:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763065.534397:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763065.534401:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763065.534409:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2992 00000100:00080000:0.0:1789763065.534418:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763065.534429:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763065.534451:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763065.534488:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504293760:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763065.534490:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504293888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763065.534532:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763065.534540:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763065.534544:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763065.534554:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2992 00000100:00080000:0.0:1789763065.534567:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763065.534573:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763065.534583:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763065.534598:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763065.534629:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504294016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763065.534648:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763065.534653:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763065.534655:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763065.534661:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2992 00000100:00080000:0.0:1789763065.534667:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763065.534674:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763065.534682:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763065.534707:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504294144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763065.534711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763065/real 1789763065] req@ffff9018c8b99f80 x1876699504293760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763065.534848:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763065.534855:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763065.534858:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763065.534868:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504293760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763065.534876:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2992 00000100:00080000:0.0:1789763065.534894:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763065.534913:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789763065.534917:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789763065.534921:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763065.534928:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763065.534944:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504293760:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763065.534953:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504294272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763065.534968:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763065.534975:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763065.534978:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763065.534985:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2992 00000100:00080000:0.0:1789763065.534991:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763065.534998:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504293888:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763065.535010:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763065.535024:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763065.535095:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504294400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763065.535113:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763065.535156:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763065/real 1789763065] req@ffff9018c8b9b480 x1876699504293888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763065.535177:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504293888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763065.535189:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763065.535194:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763065.535198:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504293888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763065.535207:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504294016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763065.535230:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763065.535248:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763065/real 1789763065] req@ffff9018c8b9b100 x1876699504294016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763065.535272:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504294016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763065.535288:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763065.535294:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763065.535301:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504294016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763065.535314:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504294144:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763065.535353:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763065.535370:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763065/real 1789763065] req@ffff9018c8b9bb80 x1876699504294144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763065.535385:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504294144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763065.535404:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763065.535408:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763065.535412:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504294144:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763065.535419:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504294272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763065.535441:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763065.535452:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763065/real 1789763065] req@ffff9018c8b98700 x1876699504294272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763065.535466:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504294272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763065.535476:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763065.535479:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763065.535483:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504294272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763065.535507:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504294400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763065.535527:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763065.535538:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763065/real 1789763065] req@ffff9018c8b99f80 x1876699504294400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763065.535551:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504294400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763065.535582:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763065.535586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763065.535591:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504294400:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763066.174281:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763067.198225:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763068.222279:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763069.246341:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763070.270284:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763070.654254:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763070.654267:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763070.654275:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763070.654292:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 2997 00000100:00080000:0.0:1789763070.654312:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763070.654325:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763070.654404:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504294528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763070.654445:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763070.654450:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763070.654453:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763070.654459:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2997 00000100:00080000:0.0:1789763070.654483:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763070.654503:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763070.654520:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763070.654553:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504294656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763070.654555:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504294528:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763070.654603:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763070.654608:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763070.654612:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763070.654621:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2997 00000100:00080000:0.0:1789763070.654627:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763070.654631:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763070.654638:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763070.654655:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504294784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763070.654668:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763070.654674:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763070.654677:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763070.654682:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 2997 00000100:00080000:0.0:1789763070.654686:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763070.654691:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789763070.654701:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789763070.654706:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763070.654722:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504294912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763070.654733:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763070.654737:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763070.654740:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763070.654744:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2997 00000100:00080000:0.0:1789763070.654748:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763070.654753:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763070.654757:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763070.654767:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504295040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763070.654777:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763070.654781:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763070.654783:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763070.654787:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 2997 00000100:00080000:0.0:1789763070.654791:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763070.654795:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763070.654801:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763070.654811:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504295168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763070.655210:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763070/real 1789763070] req@ffff9018c8b98a80 x1876699504294528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763070.655319:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504294528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763070.655353:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763070.655358:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763070.655364:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504294528:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763070.655399:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504294656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763070.655435:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763070.655455:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763070/real 1789763070] req@ffff9018c8b99880 x1876699504294656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763070.655471:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504294656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763070.655481:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763070.655485:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763070.655489:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504294656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763070.655496:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504294784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763070.655515:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763070.655526:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763070/real 1789763070] req@ffff9018c8b9b800 x1876699504294784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763070.655540:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504294784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763070.655550:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763070.655553:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763070.655557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504294784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763070.655585:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504294912:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763070.655627:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763070.655643:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763070/real 1789763070] req@ffff9018c8b98000 x1876699504294912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763070.655658:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504294912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763070.655669:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763070.655673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763070.655677:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504294912:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763070.655684:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504295040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763070.655705:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763070.655717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763070/real 1789763070] req@ffff9018c8b9aa00 x1876699504295040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763070.655731:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504295040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763070.655741:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763070.655745:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763070.655748:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504295040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763070.655755:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504295168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763070.655773:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763070.655784:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763070/real 1789763070] req@ffff9018c8b98e00 x1876699504295168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763070.655798:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504295168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763070.655808:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763070.655811:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763070.655816:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504295168:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763071.294251:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763072.318263:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763073.342146:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763074.366326:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763075.390219:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763075.774185:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763075.774196:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763075.774200:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763075.774210:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3002 00000100:00080000:2.0:1789763075.774222:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763075.774229:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763075.774271:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504295296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763075.774295:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763075.774299:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763075.774300:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763075.774304:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3002 00000100:00080000:2.0:1789763075.774306:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763075.774308:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763075.774317:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763075.774328:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504295424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763075.774336:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763075.774341:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763075.774343:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763075.774343:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504295296:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763075.774359:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3002 00000100:00080000:2.0:1789763075.774364:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763075.774367:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763075.774372:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763075.774380:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504295552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763075.774387:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763075.774390:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763075.774392:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763075.774395:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3002 00000100:00080000:2.0:1789763075.774409:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763075.774412:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763075.774416:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789763075.774431:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763075.774437:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504295680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763075.774445:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763075.774449:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763075.774450:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763075.774453:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3002 00000100:00080000:2.0:1789763075.774456:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763075.774458:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763075.774461:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763075.774468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504295808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763075.774475:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763075.774477:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763075.774479:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763075.774482:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763075/real 1789763075] req@ffff9018d0f25c00 x1876699504295296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763075.774485:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3002 00000100:00080000:2.0:1789763075.774495:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763075.774501:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763075.774508:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763075.774518:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504295936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763075.774523:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504295296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763075.774549:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763075.774553:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763075.774558:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504295296:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763075.774582:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504295424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763075.774601:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763075.774611:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763075/real 1789763075] req@ffff9018d0f27800 x1876699504295424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763075.774619:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504295424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763075.774625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763075.774627:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763075.774630:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504295424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763075.774633:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504295552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763075.774643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763075.774650:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763075/real 1789763075] req@ffff9018d0f25180 x1876699504295552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763075.774658:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504295552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763075.774663:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763075.774664:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763075.774666:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504295552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763075.774670:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504295680:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763075.774681:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763075.774687:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763075/real 1789763075] req@ffff9018d0f24380 x1876699504295680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763075.774695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504295680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763075.774700:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763075.774702:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763075.774704:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504295680:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763075.774707:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504295808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763075.774717:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763075.774722:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763075/real 1789763075] req@ffff9018d0f24e00 x1876699504295808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763075.774730:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504295808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763075.774739:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763075.774742:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763075.774746:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504295808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763075.774752:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504295936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763075.774776:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763075.774788:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763075/real 1789763075] req@ffff9018d0f24000 x1876699504295936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763075.774797:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504295936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763075.774803:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763075.774806:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763075.774808:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504295936:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763076.414329:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763077.438252:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763078.462306:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763079.486357:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763080.510279:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763080.894226:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763080.894238:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763080.894244:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763080.894256:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3007 00000100:00080000:3.0:1789763080.894271:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763080.894279:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763080.894326:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504296064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763080.894367:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763080.894373:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763080.894376:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763080.894382:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3007 00000100:00080000:3.0:1789763080.894387:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763080.894394:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763080.894412:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763080.894430:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504296064:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763080.894441:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504296192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763080.894468:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763080.894475:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763080.894479:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763080.894516:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3007 00000100:00080000:3.0:1789763080.894548:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763080.894560:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789763080.894564:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789763080.894570:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763080.894593:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504296320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763080.894607:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763080.894612:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763080.894616:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763080.894622:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3007 00000100:00080000:3.0:1789763080.894629:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763080.894638:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763080/real 1789763080] req@ffff9018d070ca80 x1876699504296064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763080.894640:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763080.894656:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763080.894678:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504296448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763080.894694:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504296064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763080.894695:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763080.894711:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763080.894717:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763080.894731:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3007 00000100:00080000:2.0:1789763080.894734:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763080.894737:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763080.894749:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789763080.894751:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763080.894762:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763080.894764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504296064:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763080.894783:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504296576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763080.894799:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763080.894804:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789763080.894809:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504296192:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789763080.894810:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763080.894827:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3007 00000100:00080000:3.0:1789763080.894833:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763080.894838:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763080.894846:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763080.894862:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763080.894868:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504296704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763080.894887:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763080/real 1789763080] req@ffff9018d070d180 x1876699504296192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763080.894916:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504296192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763080.894934:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763080.894944:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763080.894950:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504296192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763080.894962:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504296320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763080.895003:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763080.895026:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763080/real 1789763080] req@ffff9018d070e680 x1876699504296320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763080.895092:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504296320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763080.895120:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763080.895127:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763080.895133:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504296320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763080.895145:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504296448:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763080.895204:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763080.895231:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763080/real 1789763080] req@ffff9018d070f100 x1876699504296448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763080.895246:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504296448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763080.895257:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763080.895261:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763080.895265:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504296448:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763080.895273:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504296576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763080.895298:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763080.895311:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763080/real 1789763080] req@ffff9018d070f800 x1876699504296576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763080.895324:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504296576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763080.895334:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763080.895337:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763080.895342:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504296576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763080.895366:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504296704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763080.895386:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763080.895398:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763080/real 1789763080] req@ffff9018d070d500 x1876699504296704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763080.895412:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504296704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763080.895421:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763080.895424:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763080.895428:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504296704:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763081.534294:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763082.558146:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763083.582323:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763084.606260:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763085.630289:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763086.014199:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763086.014213:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763086.014220:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763086.014232:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3012 00000100:00080000:2.0:1789763086.014246:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763086.014255:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763086.014316:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504296832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763086.014378:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763086.014384:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763086.014387:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763086.014392:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3012 00000100:00080000:2.0:1789763086.014396:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763086.014401:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763086.014415:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763086.014433:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504296960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763086.014488:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763086.014499:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789763086.014502:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504296832:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763086.014503:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763086.014524:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3012 00000100:00080000:2.0:1789763086.014530:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763086.014552:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763086.014562:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763086.014582:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504297088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763086.014595:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763086.014600:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763086.014603:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763086.014608:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3012 00000100:00080000:2.0:1789763086.014632:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763086.014640:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789763086.014648:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789763086.014649:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763086.014676:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504297216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763086.014693:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763086.014698:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763086.014700:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763086.014705:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3012 00000100:00080000:2.0:1789763086.014711:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763086.014717:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763086.014723:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763086.014736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763086/real 1789763086] req@ffff9018d0f24700 x1876699504296832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763086.014744:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504297344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763086.014847:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763086.014852:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763086.014855:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763086.014862:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3012 00000100:00080000:2.0:1789763086.014868:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763086.014874:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504296832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763086.014884:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763086.014913:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789763086.014918:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763086.014929:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763086.014936:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504296832:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763086.014968:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504297472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763086.015000:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504296960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763086.015094:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763086.015115:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763086/real 1789763086] req@ffff9018d0f26680 x1876699504296960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763086.015134:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504296960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763086.015146:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763086.015151:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763086.015156:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504296960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763086.015164:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504297088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763086.015188:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763086.015200:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763086/real 1789763086] req@ffff9018d0f27480 x1876699504297088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763086.015215:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504297088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763086.015234:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763086.015239:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763086.015243:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504297088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763086.015251:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504297216:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763086.015272:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763086.015294:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763086/real 1789763086] req@ffff9018d0f25500 x1876699504297216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763086.015308:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504297216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763086.015318:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763086.015322:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763086.015325:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504297216:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763086.015332:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504297344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763086.015351:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763086.015372:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763086/real 1789763086] req@ffff9018d0f24000 x1876699504297344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763086.015396:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504297344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763086.015413:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763086.015419:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763086.015427:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504297344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763086.015440:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504297472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763086.015515:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763086.015533:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763086/real 1789763086] req@ffff9018d0f24e00 x1876699504297472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763086.015549:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504297472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763086.015560:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763086.015564:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763086.015568:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504297472:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763086.654314:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763087.678289:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763088.702346:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763089.726354:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763090.750257:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763091.134170:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763091.134186:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763091.134192:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763091.134208:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3017 00000100:00080000:2.0:1789763091.134226:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763091.134238:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763091.134299:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504297600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763091.134355:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763091.134361:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763091.134363:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763091.134369:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3017 00000100:00080000:2.0:1789763091.134373:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763091.134378:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763091.134393:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763091.134410:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504297728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763091.134450:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763091.134456:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763091.134459:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763091.134465:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3017 00000100:00080000:2.0:1789763091.134469:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763091.134475:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763091.134481:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763091.134497:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504297856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763091.134519:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763091.134519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504297600:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763091.134543:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763091.134551:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763091.134558:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3017 00000100:00080000:2.0:1789763091.134561:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763091.134566:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763091.134573:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763091.134592:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504297984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763091.134604:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763091.134610:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763091.134612:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763091.134616:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3017 00000100:00080000:2.0:1789763091.134631:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763091.134638:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763091.134645:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763091.134658:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504298112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763091.134670:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763091.134675:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763091.134679:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763091.134688:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3017 00000100:00080000:2.0:1789763091.134694:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763091.134701:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763091.134710:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763091.134715:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504297728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763091.134733:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504298240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763091.134808:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504297856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763091.134864:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504297984:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763091.134897:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504298112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763091.134922:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504298240:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763091.774275:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763092.798269:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763093.822276:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763094.206752:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789763094.206816:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763094.206832:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763094.206880:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763094.206893:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763094.206903:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763094.206913:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763094.206926:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789763094.207094:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763091/real 1789763094] req@ffff9018d0f27800 x1876699504297856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763094.207166:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504297856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763094.207192:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763094.207204:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763094.207212:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504297856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763094.207228:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763091/real 1789763094] req@ffff9018d0f25c00 x1876699504297984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763094.207251:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504297984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763094.207266:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763094.207272:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763094.207303:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504297984:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763094.207319:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763091/real 1789763094] req@ffff9018d0f26a00 x1876699504298112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763094.207345:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504298112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763094.207362:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763094.207369:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763094.207376:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504298112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763094.207418:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763091/real 1789763094] req@ffff9018d0f24a80 x1876699504298240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763094.207448:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504298240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763094.207466:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763094.207474:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763094.207480:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504298240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763094.207495:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763091/real 1789763094] req@ffff9018d0f24380 x1876699504297600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763094.207523:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504297600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763094.207540:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763094.207547:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763094.207553:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504297600:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763094.207568:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763091/real 1789763094] req@ffff9018d0f25180 x1876699504297728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763094.207601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504297728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763094.207618:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763094.207625:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763094.207632:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504297728:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763094.846213:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763095.870335:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763096.254144:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763096.254157:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763096.254162:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763096.254176:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3022 00000100:00080000:2.0:1789763096.254190:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763096.254201:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763096.254252:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504298368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763096.254289:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763096.254294:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763096.254297:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763096.254302:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3022 00000100:00080000:2.0:1789763096.254307:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763096.254311:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763096.254325:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763096.254343:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504298496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763096.254357:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763096.254362:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763096.254364:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763096.254369:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3022 00000100:00080000:2.0:1789763096.254393:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763096.254400:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763096.254406:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763096.254420:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504298624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763096.254433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763096.254442:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789763096.254444:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504298368:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763096.254446:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763096.254461:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3022 00000100:00080000:2.0:1789763096.254467:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763096.254471:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763096.254478:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763096.254492:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504298752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763096.254506:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763096.254511:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763096.254513:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763096.254517:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3022 00000100:00080000:2.0:1789763096.254532:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763096.254540:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763096.254552:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763096.254552:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763096.254573:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504298880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763096.254596:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763096.254603:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763096.254607:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763096.254616:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3022 00000100:00080000:2.0:1789763096.254625:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763096.254636:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763096/real 1789763096] req@ffff9018d0f26d80 x1876699504298368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763096.254637:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763096.254652:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763096.254684:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504299008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763096.254695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504298368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763096.254736:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763096.254743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763096.254749:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504298368:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763096.254762:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504298496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763096.254810:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763096.254839:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763096/real 1789763096] req@ffff9018d0f25f80 x1876699504298496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763096.254911:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504298496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763096.254930:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763096.254937:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763096.254945:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504298496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763096.254988:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504298624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763096.255101:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763096.255130:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763096/real 1789763096] req@ffff9018d0f27100 x1876699504298624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763096.255157:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504298624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763096.255185:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763096.255192:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763096.255199:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504298624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763096.255213:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504298752:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763096.255254:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763096.255277:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763096/real 1789763096] req@ffff9018d0f26300 x1876699504298752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763096.255744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504298752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763096.255761:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763096.255767:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763096.255774:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504298752:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763096.255788:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504298880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763096.255837:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763096.256010:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763096/real 1789763096] req@ffff9018d0f25180 x1876699504298880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763096.256080:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504298880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763096.256098:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763096.256105:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763096.256114:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504298880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763096.256126:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504299008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763096.256172:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763096.256198:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763096/real 1789763096] req@ffff9018d0f24380 x1876699504299008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763096.256222:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504299008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763096.256238:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763096.256244:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763096.256251:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504299008:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763096.894284:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763097.918335:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763098.942309:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763099.966256:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763100.990322:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763101.374229:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763101.374247:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763101.374252:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763101.374269:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3027 00000100:00080000:3.0:1789763101.374286:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763101.374297:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763101.374408:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504299136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763101.374450:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763101.374456:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763101.374458:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763101.374464:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3027 00000100:00080000:3.0:1789763101.374469:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763101.374476:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763101.374509:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763101.374528:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504299264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763101.374541:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763101.374547:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763101.374550:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763101.374556:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3027 00000100:00080000:3.0:1789763101.374560:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763101.374565:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763101.374571:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763101.374584:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504299392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763101.374598:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504299136:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763101.374603:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763101.374615:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763101.374618:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763101.374624:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3027 00000100:00080000:3.0:1789763101.374628:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763101.374633:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763101.374640:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763101.374658:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504299520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763101.374672:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763101.374678:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763101.374682:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763101.374686:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3027 00000100:00080000:3.0:1789763101.374704:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763101.374709:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763101.374716:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763101.374724:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763101.374740:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504299648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763101.374758:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763101.374762:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763101.374765:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763101.374770:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3027 00000100:00080000:3.0:1789763101.374777:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763101.374782:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763101.374788:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763101.374797:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763101/real 1789763101] req@ffff9018d070dc00 x1876699504299136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763101.374827:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504299776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763101.374854:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504299136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763101.374871:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763101.374877:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763101.374883:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504299136:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763101.374893:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504299264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763101.374923:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763101.374937:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763101/real 1789763101] req@ffff9018d070c000 x1876699504299264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763101.374952:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504299264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763101.374962:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763101.374967:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763101.374970:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504299264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763101.375007:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504299392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763101.375027:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763101.375039:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763101/real 1789763101] req@ffff9018d070e300 x1876699504299392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763101.375084:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504299392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763101.375094:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763101.375099:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763101.375103:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504299392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763101.375110:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504299520:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763101.375132:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763101.375145:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763101/real 1789763101] req@ffff9018d070d500 x1876699504299520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763101.375158:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504299520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763101.375168:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763101.375171:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763101.375174:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504299520:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763101.375181:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504299648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763101.375200:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763101.375230:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763101/real 1789763101] req@ffff9018d070f800 x1876699504299648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763101.375244:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504299648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763101.375254:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763101.375257:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763101.375261:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504299648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763101.375267:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504299776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763101.375288:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763101.375298:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763101/real 1789763101] req@ffff9018d070f100 x1876699504299776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763101.375312:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504299776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763101.375322:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763101.375325:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763101.375329:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504299776:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763102.014286:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763103.038259:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763104.062305:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763105.086344:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763106.110326:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763106.494195:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763106.494211:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763106.494220:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763106.494243:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3033 00000100:00080000:3.0:1789763106.494266:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763106.494282:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763106.494379:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504299904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763106.494438:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763106.494444:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763106.494447:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763106.494454:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3033 00000100:00080000:3.0:1789763106.494460:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763106.494466:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763106.494485:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763106.494515:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504300032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763106.494531:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763106.494538:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763106.494541:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763106.494546:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3033 00000100:00080000:3.0:1789763106.494553:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763106.494564:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504299904:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763106.494566:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763106.494579:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763106.494595:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504300160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763106.494611:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763106.494616:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763106.494619:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763106.494627:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3033 00000100:00080000:3.0:1789763106.494749:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763106.494759:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763106.494771:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789763106.494788:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763106.494795:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504300288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763106.494815:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763106.494877:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763106/real 1789763106] req@ffff9018d070e680 x1876699504299904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763106.494919:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504299904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763106.494937:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763106.494943:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763106.494949:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504299904:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763106.494961:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504300032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763106.495002:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763106.495024:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763106/real 1789763106] req@ffff9018d070d180 x1876699504300032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763106.495076:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763106.495080:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763106.495165:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504300032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763106.495173:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3033 00000100:00080000:3.0:1789763106.495187:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763106.495193:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763106.495201:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789763106.495211:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000008:00100000:3.0:1789763106.495217:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763106.495226:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504300032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763106.495251:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504300416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763106.495280:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763106.495281:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504300160:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789763106.495326:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763106.495333:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763106.495344:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3033 00000100:00080000:3.0:1789763106.495355:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763106.495367:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763106.495383:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763106.495393:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763106/real 1789763106] req@ffff9018d070ca80 x1876699504300160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789763106.495397:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763106.495422:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504300160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763106.495430:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504300544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763106.495488:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763106.495505:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763106.495516:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504300160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763106.495535:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504300288:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763106.495573:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763106.495590:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763106/real 1789763106] req@ffff9018d070fb80 x1876699504300288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763106.495621:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504300288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763106.495646:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763106.495653:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763106.495660:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504300288:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763106.495691:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504300416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763106.495735:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763106.495754:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763106/real 1789763106] req@ffff9018d070f480 x1876699504300416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763106.495779:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504300416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763106.495790:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763106.495794:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763106.495798:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504300416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763106.495806:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504300544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763106.495840:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763106.495862:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763106/real 1789763106] req@ffff9018d070ed80 x1876699504300544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763106.495888:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504300544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763106.495905:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763106.495912:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763106.495918:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504300544:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763107.134249:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763108.158317:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763109.182272:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763110.206254:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763111.230309:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763111.614152:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763111.614166:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763111.614171:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763111.614185:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3038 00000100:00080000:3.0:1789763111.614199:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763111.614208:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763111.614258:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504300672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763111.614320:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763111.614325:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763111.614328:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763111.614334:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3038 00000100:00080000:3.0:1789763111.614339:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763111.614345:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763111.614358:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763111.614379:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504300800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763111.614392:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763111.614397:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763111.614400:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763111.614405:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3038 00000100:00080000:3.0:1789763111.614410:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763111.614419:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504300672:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763111.614421:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763111.614441:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763111.614459:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504300928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763111.614475:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763111.614479:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763111.614482:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763111.614487:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3038 00000100:00080000:3.0:1789763111.614492:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763111.614496:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763111.614504:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789763111.614525:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763111.614527:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504301056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763111.614545:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763111.614550:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763111.614554:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763111.614558:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3038 00000100:00080000:3.0:1789763111.614563:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763111.614568:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763111.614575:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763111.614589:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763111/real 1789763111] req@ffff9018d070ce00 x1876699504300672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763111.614600:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504301184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763111.614631:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504300672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763111.614633:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763111.614656:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763111.614666:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763111.614688:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3038 00000100:00080000:0.0:1789763111.614689:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763111.614703:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789763111.614704:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763111.614715:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504300672:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763111.614719:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763111.614735:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763111.614737:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504300800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763111.614764:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504301312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763111.614780:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763111.614804:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763111/real 1789763111] req@ffff9018d070df80 x1876699504300800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763111.614822:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504300800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763111.614833:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763111.614837:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763111.614841:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504300800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763111.614872:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504300928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763111.614893:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763111.614905:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763111/real 1789763111] req@ffff9018d070d880 x1876699504300928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763111.614918:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504300928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763111.614931:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763111.614937:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763111.614944:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504300928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763111.614957:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504301056:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763111.614993:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763111.615017:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763111/real 1789763111] req@ffff9018d070ed80 x1876699504301056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763111.615094:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504301056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763111.615106:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763111.615110:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763111.615114:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504301056:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763111.615122:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504301184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763111.615152:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763111.615166:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763111/real 1789763111] req@ffff9018d070f480 x1876699504301184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763111.615181:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504301184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763111.615191:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763111.615194:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763111.615198:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504301184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763111.615205:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504301312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763111.615223:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763111.615233:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763111/real 1789763111] req@ffff9018d070fb80 x1876699504301312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763111.615248:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504301312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763111.615257:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763111.615285:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763111.615290:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504301312:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763112.254162:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763113.278331:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763114.302266:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763115.326207:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763116.350244:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763116.734104:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763116.734116:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763116.734121:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763116.734132:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3043 00000100:00080000:3.0:1789763116.734146:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763116.734154:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763116.734201:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504301440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763116.734242:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763116.734246:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763116.734247:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763116.734252:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3043 00000100:00080000:3.0:1789763116.734255:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763116.734258:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763116.734268:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763116.734280:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504301568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763116.734293:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763116.734295:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763116.734296:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763116.734299:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3043 00000100:00080000:3.0:1789763116.734302:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763116.734305:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763116.734308:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763116.734315:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504301696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763116.734321:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763116.734324:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763116.734326:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763116.734332:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3043 00000100:00080000:3.0:1789763116.734334:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763116.734337:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763116.734340:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763116.734347:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504301824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763116.734354:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763116.734357:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763116.734358:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763116.734360:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3043 00000100:00080000:3.0:1789763116.734363:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763116.734366:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763116.734370:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763116.734377:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504301952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763116.734385:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763116.734388:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763116.734391:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763116.734396:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3043 00000100:00080000:3.0:1789763116.734400:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763116.734407:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763116.734408:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504301440:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789763116.734411:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763116.734426:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504302080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763116.734531:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763116.734607:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763116/real 1789763116] req@ffff9018d070ca80 x1876699504301440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763116.734633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504301440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763116.734652:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763116.734658:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763116.734664:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504301440:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763116.734675:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504301568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763116.734705:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763116.734721:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763116/real 1789763116] req@ffff9018d070d180 x1876699504301568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763116.734737:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504301568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763116.734756:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763116.734760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763116.734764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504301568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763116.734772:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504301696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763116.734791:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763116.734802:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763116/real 1789763116] req@ffff9018d070e680 x1876699504301696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763116.734816:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504301696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763116.734826:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763116.734830:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763116.734833:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504301696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763116.734841:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504301824:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763116.734859:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763116.734870:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763116/real 1789763116] req@ffff9018d070f100 x1876699504301824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763116.734883:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504301824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763116.734893:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763116.734897:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763116.734901:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504301824:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763116.734938:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504301952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763116.734959:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763116.734972:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763116/real 1789763116] req@ffff9018d070f800 x1876699504301952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763116.734986:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504301952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763116.734995:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763116.734999:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763116.735002:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504301952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763116.735009:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504302080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763116.735028:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763116.735118:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763116/real 1789763116] req@ffff9018d070d500 x1876699504302080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763116.735135:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504302080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763116.735145:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763116.735149:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763116.735153:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504302080:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763117.374240:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763118.398320:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763119.422242:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763120.446280:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763121.470286:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763121.854178:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763121.854218:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763121.854225:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763121.854254:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3048 00000100:00080000:1.0:1789763121.854270:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763121.854281:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763121.854340:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504302208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763121.854388:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763121.854393:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763121.854396:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763121.854403:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3048 00000100:00080000:1.0:1789763121.854407:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763121.854413:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763121.854428:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763121.854459:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504302336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763121.854476:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763121.854482:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504302208:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763121.854484:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763121.854492:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763121.854502:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3048 00000100:00080000:1.0:1789763121.854507:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763121.854511:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763121.854517:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763121.854536:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504302464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763121.854549:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763121.854554:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763121.854557:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763121.854562:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3048 00000100:00080000:1.0:1789763121.854580:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763121.854586:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763121.854593:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763121.854609:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504302592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763121.854625:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763121.854630:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763121.854632:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763121.854637:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3048 00000100:00080000:1.0:1789763121.854642:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763121.854645:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763121.854662:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763121.854671:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763121.854692:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504302720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763121.854708:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763121.854714:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763121.854718:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763121.854725:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763121/real 1789763121] req@ffff9018c66a5c00 x1876699504302208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763121.854732:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3048 00000100:00080000:1.0:1789763121.854743:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763121.854755:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763121.854762:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763121.854777:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504302848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763121.854795:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504302208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763121.854828:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763121.854839:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763121.854849:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504302208:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763121.854896:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504302336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763121.854934:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763121.854953:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763121/real 1789763121] req@ffff9018c66a6d80 x1876699504302336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763121.854970:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504302336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763121.854981:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763121.854985:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763121.854990:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504302336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763121.854997:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504302464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763121.855016:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763121.855029:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763121/real 1789763121] req@ffff9018c66a5f80 x1876699504302464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763121.855105:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504302464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763121.855125:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763121.855130:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763121.855134:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504302464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763121.855142:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504302592:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763121.855168:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763121.855182:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763121/real 1789763121] req@ffff9018c66a4000 x1876699504302592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763121.855228:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504302592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763121.855250:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763121.855254:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763121.855258:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504302592:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763121.855266:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504302720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763121.855293:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763121.855307:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763121/real 1789763121] req@ffff9018c66a7b80 x1876699504302720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763121.855321:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504302720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763121.855330:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763121.855333:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763121.855338:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504302720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763121.855345:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504302848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763121.855365:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763121.855375:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763121/real 1789763121] req@ffff9018c66a7800 x1876699504302848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763121.855389:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504302848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763121.855399:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763121.855402:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763121.855406:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504302848:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763122.494284:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763123.518317:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763124.542234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763125.566326:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763126.590275:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763126.974290:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763126.974305:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763126.974312:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763126.974326:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3053 00000100:00080000:0.0:1789763126.974343:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763126.974353:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763126.974413:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504302976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763126.974451:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763126.974455:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763126.974458:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763126.974462:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3053 00000100:00080000:0.0:1789763126.974465:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763126.974471:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763126.974485:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763126.974510:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504303104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763126.974521:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763126.974525:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763126.974527:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763126.974531:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3053 00000100:00080000:0.0:1789763126.974534:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763126.974538:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763126.974542:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763126.974561:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504303232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763126.974571:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504302976:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763126.974575:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763126.974599:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763126.974602:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763126.974609:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3053 00000100:00080000:0.0:1789763126.974612:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763126.974616:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763126.974622:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763126.974637:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504303360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763126.974647:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763126.974652:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763126.974654:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763126.974658:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3053 00000100:00080000:0.0:1789763126.974682:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763126.974687:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763126.974691:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763126.974703:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504303488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763126.974716:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763126.974720:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763126.974725:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763126.974727:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763126.974742:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3053 00000100:00080000:0.0:1789763126.974756:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763126.974762:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763126.974767:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763126.974789:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504303616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763126.974802:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763126/real 1789763126] req@ffff9018c8b9a680 x1876699504302976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763126.974833:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504302976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763126.974855:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763126.974863:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763126.974870:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504302976:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763126.974890:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504303104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763126.974935:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763126.974953:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763126/real 1789763126] req@ffff9018c8b99c00 x1876699504303104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763126.974969:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504303104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763126.974979:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763126.974983:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763126.974987:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504303104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763126.975017:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504303232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763126.975039:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763126.975050:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763126/real 1789763126] req@ffff9018c8b98380 x1876699504303232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763126.975124:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504303232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763126.975132:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763126.975134:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763126.975137:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504303232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763126.975143:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504303360:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763126.975204:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763126.975219:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763126/real 1789763126] req@ffff9018c8b99500 x1876699504303360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763126.975233:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504303360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763126.975243:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763126.975247:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763126.975250:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504303360:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763126.975257:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504303488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763126.975277:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763126.975287:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763126/real 1789763126] req@ffff9018c8b9ad80 x1876699504303488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763126.975300:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504303488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763126.975309:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763126.975312:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763126.975315:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504303488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763126.975322:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504303616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763126.975339:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763126.975348:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763126/real 1789763126] req@ffff9018c8b9a300 x1876699504303616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763126.975361:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504303616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763126.975377:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763126.975381:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763126.975384:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504303616:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763127.614294:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763128.638170:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763129.662203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763130.686280:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763131.710303:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763132.094225:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763132.094242:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763132.094249:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763132.094264:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3058 00000100:00080000:1.0:1789763132.094279:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763132.094288:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763132.094338:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504303744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763132.094379:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763132.094384:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763132.094387:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763132.094392:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3058 00000100:00080000:1.0:1789763132.094397:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763132.094402:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763132.094417:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763132.094434:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504303872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763132.094450:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763132.094457:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504303744:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763132.094458:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763132.094465:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763132.094478:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3058 00000100:00080000:1.0:1789763132.094482:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763132.094486:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763132.094492:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763132.094510:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504304000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763132.094536:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763132.094542:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763132.094545:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763132.094549:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3058 00000100:00080000:1.0:1789763132.094556:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763132.094565:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763132.094568:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763132.094579:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763132.094597:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504304128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763132.094621:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763132.094626:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763132.094629:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763132.094635:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3058 00000100:00080000:1.0:1789763132.094640:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763132.094645:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763132/real 1789763132] req@ffff9018c66a5180 x1876699504303744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763132.094650:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763132.094660:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763132.094680:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504304256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763132.094695:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763132.094697:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504303744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763132.094706:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763132.094713:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763132.094726:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3058 00000100:00080000:2.0:1789763132.094730:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763132.094732:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763132.094743:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789763132.094748:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:1.0:1789763132.094753:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763132.094767:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504304384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763132.094779:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504303744:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763132.094822:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504303872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763132.094857:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763132.094875:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763132/real 1789763132] req@ffff9018c66a6300 x1876699504303872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763132.094892:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504303872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763132.094903:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763132.094908:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763132.094912:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504303872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763132.094919:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504304000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763132.094939:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763132.094951:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763132/real 1789763132] req@ffff9018c66a4700 x1876699504304000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763132.094965:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504304000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763132.094975:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763132.094978:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763132.094982:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504304000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763132.094989:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504304128:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763132.095007:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763132.095018:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763132/real 1789763132] req@ffff9018c66a6680 x1876699504304128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763132.095090:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504304128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763132.095103:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763132.095106:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763132.095142:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504304128:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763132.095152:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504304256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763132.095183:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763132.095199:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763132/real 1789763132] req@ffff9018c66a5500 x1876699504304256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763132.095214:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504304256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763132.095224:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763132.095227:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763132.095232:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504304256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763132.095238:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504304384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763132.095256:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763132.095267:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763132/real 1789763132] req@ffff9018c66a4e00 x1876699504304384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763132.095280:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504304384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763132.095290:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763132.095293:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763132.095296:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504304384:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763132.734280:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763133.758312:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763134.782261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763135.806173:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763136.830258:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763137.214107:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763137.214116:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763137.214120:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763137.214133:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3063 00000100:00080000:0.0:1789763137.214142:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763137.214147:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763137.214179:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504304512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763137.214201:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763137.214204:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763137.214205:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763137.214208:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3063 00000100:00080000:0.0:1789763137.214210:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763137.214213:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763137.214220:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763137.214229:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504304640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763137.214237:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763137.214240:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763137.214242:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763137.214245:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3063 00000100:00080000:0.0:1789763137.214249:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763137.214253:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763137.214256:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763137.214264:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504304768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763137.214272:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763137.214274:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763137.214275:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763137.214279:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3063 00000100:00080000:0.0:1789763137.214282:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763137.214288:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763137.214298:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763137.214299:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504304512:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763137.214311:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504304896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763137.214322:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763137.214324:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763137.214325:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763137.214327:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3063 00000100:00080000:0.0:1789763137.214330:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763137.214334:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763137.214336:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763137.214347:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504305024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763137.214353:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763137.214355:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763137.214356:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763137.214360:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3063 00000100:00080000:0.0:1789763137.214362:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763137.214366:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763137.214369:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763137.214377:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504305152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763137.214419:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763137.214477:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763137/real 1789763137] req@ffff9018c8b9a300 x1876699504304512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763137.214503:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504304512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763137.214519:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763137.214525:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763137.214531:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504304512:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763137.214541:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504304640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763137.214567:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763137.214581:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763137/real 1789763137] req@ffff9018c8b9ad80 x1876699504304640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763137.214597:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504304640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763137.214607:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763137.214612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763137.214616:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504304640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763137.214649:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504304768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763137.214674:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763137.214685:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763137/real 1789763137] req@ffff9018c8b99500 x1876699504304768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763137.214700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504304768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763137.214710:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763137.214713:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763137.214717:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504304768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763137.214724:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504304896:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763137.214744:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763137.214772:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763137/real 1789763137] req@ffff9018c8b98380 x1876699504304896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763137.214787:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504304896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763137.214798:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763137.214802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763137.214806:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504304896:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763137.214813:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504305024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763137.214834:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763137.214845:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763137/real 1789763137] req@ffff9018c8b99c00 x1876699504305024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763137.214879:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504305024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763137.214890:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763137.214893:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763137.214897:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504305024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763137.214904:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504305152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763137.214925:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763137.214936:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763137/real 1789763137] req@ffff9018c8b9a680 x1876699504305152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763137.214950:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504305152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763137.214959:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763137.214963:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763137.214966:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504305152:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763137.854371:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763138.878287:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763139.902243:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763140.926310:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763141.950325:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763142.334306:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763142.334321:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763142.334329:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763142.334350:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3068 00000100:00080000:3.0:1789763142.334372:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763142.334384:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763142.334449:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504305280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763142.334488:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763142.334496:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763142.334499:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763142.334507:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3068 00000100:00080000:3.0:1789763142.334513:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763142.334519:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763142.334538:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763142.334540:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504305280:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763142.334574:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504305408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763142.334594:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763142.334601:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763142.334606:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763142.334618:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3068 00000100:00080000:3.0:1789763142.334624:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763142.334631:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763142.334642:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763142.334659:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504305536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763142.334672:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763142.334678:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763142.334684:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763142.334688:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763142.334695:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3068 00000100:00080000:3.0:1789763142.334703:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763142.334709:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763142.334721:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763142.334740:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504305664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763142.334757:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763142.334762:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763142.334765:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763142.334773:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3068 00000100:00080000:3.0:1789763142.334778:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763142.334784:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763142.334792:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763142.334794:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763142/real 1789763142] req@ffff9018d070e300 x1876699504305280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763142.334814:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504305792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763142.334832:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763142.334837:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763142.334841:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763142.334849:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3068 00000100:00100000:0.0:1789763142.334851:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504305280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763142.334854:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763142.334866:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789763142.334886:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763142.334896:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763142.334904:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504305280:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763142.334954:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504305408:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789763142.334991:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763142.335012:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763142.335016:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504305920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763142.335127:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763142/real 1789763142] req@ffff9018d070c000 x1876699504305408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763142.335165:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504305408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763142.335183:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763142.335190:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763142.335195:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504305408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763142.335207:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504305536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763142.335243:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763142.335261:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763142/real 1789763142] req@ffff9018d070dc00 x1876699504305536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763142.335278:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504305536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763142.335289:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763142.335293:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763142.335297:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504305536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763142.335305:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504305664:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763142.335330:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763142.335353:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763142/real 1789763142] req@ffff9018d070d500 x1876699504305664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763142.335388:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504305664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763142.335411:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763142.335423:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763142.335434:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504305664:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763142.335449:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504305792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763142.335513:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763142.335544:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763142/real 1789763142] req@ffff9018d070f800 x1876699504305792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763142.335563:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504305792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763142.335575:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763142.335580:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763142.335584:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504305792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763142.335613:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504305920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763142.335641:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763142.335655:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763142/real 1789763142] req@ffff9018d070f100 x1876699504305920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763142.335670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504305920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763142.335680:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763142.335684:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763142.335688:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504305920:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763142.974328:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763143.998372:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763145.022234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763146.046276:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763147.070243:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763147.454221:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763147.454235:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763147.454240:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763147.454252:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3074 00000100:00080000:1.0:1789763147.454285:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763147.454293:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763147.454358:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504306048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763147.454399:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763147.454403:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763147.454406:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763147.454411:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3074 00000100:00080000:1.0:1789763147.454416:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763147.454421:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763147.454435:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763147.454453:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504306176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763147.454469:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763147.454476:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789763147.454480:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504306048:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763147.454481:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763147.454513:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3074 00000100:00080000:1.0:1789763147.454518:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763147.454522:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763147.454528:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763147.454545:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504306304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763147.454556:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763147.454576:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763147.454579:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763147.454583:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3074 00000100:00080000:1.0:1789763147.454597:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763147.454612:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763147.454613:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763147.454624:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763147.454640:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504306432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763147.454650:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763147.454654:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763147.454656:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763147.454660:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3074 00000100:00080000:1.0:1789763147.454663:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763147.454668:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763147.454676:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763147/real 1789763147] req@ffff9018c66a4380 x1876699504306048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789763147.454676:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763147.454694:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504306560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763147.454709:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763147.454713:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763147.454715:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763147.454719:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504306048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763147.454725:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3074 00000100:00080000:1.0:1789763147.454735:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763147.454743:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763147.454745:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789763147.454752:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:1.0:1789763147.454759:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763147.454762:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504306048:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763147.454786:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504306688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763147.454808:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504306176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763147.454846:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763147.454862:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763147/real 1789763147] req@ffff9018c66a6a00 x1876699504306176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763147.454875:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504306176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763147.454883:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763147.454887:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763147.454890:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504306176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763147.454896:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504306304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763147.454912:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763147.454945:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763147/real 1789763147] req@ffff9018c66a7480 x1876699504306304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763147.454969:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504306304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763147.454987:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763147.454993:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763147.455024:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504306304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763147.455092:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504306432:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763147.455153:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763147.455175:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763147/real 1789763147] req@ffff9018c66a4a80 x1876699504306432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763147.455191:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504306432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763147.455202:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763147.455206:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763147.455210:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504306432:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763147.455218:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504306560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763147.455242:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763147.455254:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763147/real 1789763147] req@ffff9018c66a7100 x1876699504306560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763147.455285:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504306560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763147.455297:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763147.455302:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763147.455307:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504306560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763147.455356:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504306688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763147.455397:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763147.455417:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763147/real 1789763147] req@ffff9018c66a5880 x1876699504306688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763147.455433:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504306688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763147.455444:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763147.455448:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763147.455452:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504306688:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763148.094250:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763149.118238:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763150.142253:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763151.166258:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763152.190313:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763152.574200:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763152.574214:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763152.574220:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763152.574233:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3079 00000100:00080000:3.0:1789763152.574246:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763152.574256:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763152.574304:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504306816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763152.574344:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763152.574349:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763152.574352:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763152.574357:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3079 00000100:00080000:3.0:1789763152.574361:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763152.574366:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763152.574379:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763152.574394:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504306944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763152.574407:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763152.574412:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763152.574415:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763152.574420:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3079 00000100:00080000:3.0:1789763152.574424:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763152.574429:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763152.574434:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763152.574457:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504306816:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763152.574460:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504307072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763152.574501:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763152.574508:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763152.574513:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763152.574522:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3079 00000800:00000100:0.0:1789763152.574548:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763152.574553:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763152.574565:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763152.574578:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763152.574607:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504307200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763152.574621:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763152/real 1789763152] req@ffff9018d070e680 x1876699504306816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763152.574642:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763152.574653:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789763152.574660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504306816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763152.574662:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763152.574681:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3079 00000100:00080000:0.0:1789763152.574690:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763152.574691:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763152.574708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789763152.574710:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763152.574723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504306816:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789763152.574725:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763152.574742:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504306944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763152.574756:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504307328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763152.574773:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763152.574778:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1789763152.574786:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763152.574787:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763152.574800:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3079 00000100:00080000:3.0:1789763152.574808:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763152.574815:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763152/real 1789763152] req@ffff9018d070d180 x1876699504306944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763152.574821:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763152.574835:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763152.574847:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504306944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763152.574857:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504307456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763152.574871:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763152.574887:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763152.574895:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504306944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763152.574931:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504307072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763152.574956:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763152.575006:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763152/real 1789763152] req@ffff9018d070ca80 x1876699504307072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763152.575041:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504307072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763152.575094:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763152.575100:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763152.575104:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504307072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763152.575112:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504307200:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763152.575140:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763152.575154:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763152/real 1789763152] req@ffff9018d070fb80 x1876699504307200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763152.575168:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504307200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763152.575177:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763152.575181:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763152.575184:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504307200:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763152.575191:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504307328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763152.575208:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763152.575218:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763152/real 1789763152] req@ffff9018d070f480 x1876699504307328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763152.575232:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504307328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763152.575241:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763152.575244:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763152.575247:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504307328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763152.575253:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504307456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763152.575284:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763152.575303:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763152/real 1789763152] req@ffff9018d070ed80 x1876699504307456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763152.575326:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504307456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763152.575341:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763152.575347:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763152.575353:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504307456:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763153.214330:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763154.238426:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763155.262258:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763156.286261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763157.310289:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763157.694334:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763157.694350:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763157.694356:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763157.694372:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3084 00000100:00080000:1.0:1789763157.694390:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763157.694402:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763157.694470:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504307584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763157.694516:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763157.694522:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763157.694525:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763157.694531:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3084 00000100:00080000:1.0:1789763157.694535:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763157.694540:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763157.694556:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763157.694583:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504307712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763157.694606:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763157.694615:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789763157.694620:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504307584:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763157.694622:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763157.694643:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3084 00000100:00080000:1.0:1789763157.694654:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763157.694663:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763157.694674:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763157.694701:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504307840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763157.694714:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763157.694720:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763157.694724:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763157.694729:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3084 00000100:00080000:1.0:1789763157.694734:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763157.694740:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763157.694747:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763157.694763:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504307968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763157.694777:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763157.694783:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763157.694787:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763157.694792:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3084 00000100:00080000:1.0:1789763157.694799:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763157.694807:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763157.694814:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504307712:192.168.203.158@tcp:38:kworker.0 00000008:00100000:1.0:1789763157.694819:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763157.694847:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504308096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763157.694872:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763157.694880:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504307840:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789763157.694882:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763157.694890:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763157.694903:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3084 00000100:00080000:1.0:1789763157.694909:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763157.694918:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763157.694952:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763157.694964:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504307968:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763157.694976:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504308224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763157.695018:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504308096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763157.695168:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504308224:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763158.334364:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763159.358210:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763160.382285:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763160.766433:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789763160.766490:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763160.766512:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763160.766561:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763160.766573:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763160.766582:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763160.766591:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763160.766600:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789763160.766729:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763157/real 1789763160] req@ffff9018c66a6a00 x1876699504308096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763160.766798:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504308096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763160.766822:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763160.766833:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763160.766841:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504308096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763160.766859:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763157/real 1789763160] req@ffff9018c66a4380 x1876699504308224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763160.766931:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504308224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763160.766952:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763160.766961:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763160.766969:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504308224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763160.766985:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763157/real 1789763160] req@ffff9018c66a7100 x1876699504307712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763160.767012:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504307712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763160.767029:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763160.767147:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763160.767156:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504307712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763160.767264:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763157/real 1789763160] req@ffff9018c66a4a80 x1876699504307840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763160.767306:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504307840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763160.767322:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763160.767329:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763160.767337:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504307840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763160.767352:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763157/real 1789763160] req@ffff9018c66a7480 x1876699504307968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763160.767386:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504307968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763160.767402:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763160.767407:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763160.767414:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504307968:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763160.767430:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763157/real 1789763160] req@ffff9018c66a5880 x1876699504307584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763160.767456:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504307584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763160.767474:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763160.767484:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763160.767492:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504307584:192.168.203.158@tcp:250:kworker.0 00010000:00010000:2.0:1789763161.406329:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763162.430248:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763162.814190:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763162.814203:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763162.814207:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763162.814220:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3089 00000100:00080000:2.0:1789763162.814232:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763162.814241:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763162.814287:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504308352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763162.814316:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763162.814320:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763162.814321:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763162.814325:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3089 00000100:00080000:2.0:1789763162.814328:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763162.814331:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763162.814342:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763162.814351:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504308480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763162.814359:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763162.814362:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763162.814363:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763162.814366:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3089 00000100:00080000:2.0:1789763162.814368:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763162.814370:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763162.814373:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763162.814380:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504308608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763162.814386:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763162.814389:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763162.814391:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763162.814394:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3089 00000100:00080000:2.0:1789763162.814396:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763162.814399:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763162.814403:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763162.814411:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504308736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763162.814425:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763162.814429:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763162.814431:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763162.814440:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3089 00000100:00080000:2.0:1789763162.814445:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763162.814451:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763162.814466:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763162.814486:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504308864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763162.814494:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504308352:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763162.814505:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763162.814514:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763162.814516:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763162.814523:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3089 00000100:00080000:2.0:1789763162.814529:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763162.814534:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763162.814541:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763162.814559:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504308992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763162.814624:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763162.814699:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763162/real 1789763162] req@ffff9018d0f24a80 x1876699504308352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763162.814736:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504308352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763162.814758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763162.814766:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763162.814788:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504308352:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763162.814804:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504308480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763162.814853:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763162.814881:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763162/real 1789763162] req@ffff9018d0f26a00 x1876699504308480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763162.814957:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504308480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763162.814976:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763162.814982:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763162.814990:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504308480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763162.815004:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504308608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763162.815202:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763162.815237:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763162/real 1789763162] req@ffff9018d0f25c00 x1876699504308608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763162.815268:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504308608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763162.815289:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763162.815297:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763162.815303:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504308608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763162.815318:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504308736:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763162.815361:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763162.815386:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763162/real 1789763162] req@ffff9018d0f27800 x1876699504308736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763162.815411:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504308736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763162.815427:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763162.815434:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763162.815442:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504308736:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763162.815508:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504308864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763162.815558:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763162.815583:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763162/real 1789763162] req@ffff9018d0f24e00 x1876699504308864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763162.815634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504308864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763162.815650:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763162.815656:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763162.815663:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504308864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763162.815674:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504308992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763162.815729:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763162.815768:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763162/real 1789763162] req@ffff9018d0f24000 x1876699504308992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763162.815792:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504308992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763162.815806:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763162.815813:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763162.815831:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504308992:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763163.454301:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763164.478305:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763165.502286:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763166.526246:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763167.550223:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763167.934161:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763167.934176:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763167.934198:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763167.934209:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3094 00000100:00080000:0.0:1789763167.934221:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763167.934229:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763167.934274:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504309120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763167.934302:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763167.934306:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763167.934309:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763167.934313:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3094 00000100:00080000:0.0:1789763167.934317:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763167.934322:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763167.934334:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763167.934339:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504309120:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763167.934347:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504309248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763167.934358:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763167.934362:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763167.934363:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763167.934367:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3094 00000100:00080000:0.0:1789763167.934370:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763167.934373:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763167.934378:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763167.934387:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504309376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763167.934395:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763167.934406:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763167.934408:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763167.934411:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3094 00000100:00080000:0.0:1789763167.934415:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763167.934419:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789763167.934423:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789763167.934424:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763167.934434:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504309504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763167.934443:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763167.934446:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763167.934448:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763167.934451:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3094 00000100:00080000:0.0:1789763167.934454:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763167.934458:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763167.934462:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763167.934471:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504309632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763167.934471:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763167/real 1789763167] req@ffff9018c8b98e00 x1876699504309120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763167.934481:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763167.934484:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763167.934486:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763167.934489:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3094 00000100:00100000:2.0:1789763167.934492:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504309120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763167.934493:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763167.934496:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763167.934501:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763167.934504:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763167.934509:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763167.934510:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504309760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763167.934514:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504309120:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763167.934541:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504309248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763167.934566:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763167.934579:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763167/real 1789763167] req@ffff9018c8b9aa00 x1876699504309248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763167.934588:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504309248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763167.934595:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763167.934598:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763167.934600:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504309248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763167.934604:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504309376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763167.934620:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763167.934627:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763167/real 1789763167] req@ffff9018c8b98000 x1876699504309376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763167.934634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504309376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763167.934640:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763167.934641:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763167.934643:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504309376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763167.934647:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504309504:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763167.934658:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763167.934664:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763167/real 1789763167] req@ffff9018c8b9b800 x1876699504309504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763167.934671:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504309504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763167.934676:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763167.934678:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763167.934680:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504309504:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763167.934683:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504309632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763167.934694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763167.934700:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763167/real 1789763167] req@ffff9018c8b99880 x1876699504309632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763167.934707:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504309632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763167.934712:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763167.934714:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763167.934716:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504309632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763167.934719:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504309760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763167.934729:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763167.934750:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763167/real 1789763167] req@ffff9018c8b98a80 x1876699504309760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763167.934764:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504309760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763167.934773:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763167.934777:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763167.934781:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504309760:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763168.574267:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763169.598300:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763170.622266:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763171.646243:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763172.670274:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763173.054213:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763173.054229:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763173.054234:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763173.054247:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3099 00000100:00080000:1.0:1789763173.054260:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763173.054269:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763173.054315:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504309888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763173.054352:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763173.054358:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763173.054360:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763173.054365:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3099 00000100:00080000:1.0:1789763173.054370:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763173.054374:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763173.054389:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763173.054405:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504310016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763173.054421:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763173.054429:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789763173.054432:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504309888:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763173.054433:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763173.054449:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3099 00000100:00080000:1.0:1789763173.054453:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763173.054458:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763173.054464:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763173.054481:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504310144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763173.054493:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763173.054498:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763173.054500:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763173.054505:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3099 00000100:00080000:1.0:1789763173.054509:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763173.054514:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763173.054520:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789763173.054532:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763173.054539:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504310272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763173.054557:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763173.054561:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763173.054564:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763173.054570:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3099 00000100:00080000:1.0:1789763173.054576:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763173.054581:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763173.054588:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763173.054591:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763173/real 1789763173] req@ffff9018c66a4e00 x1876699504309888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763173.054607:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504310400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763173.054621:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763173.054624:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504309888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763173.054626:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763173.054633:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763173.054651:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3099 00000100:00080000:1.0:1789763173.054658:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763173.054659:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763173.054666:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789763173.054674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:1.0:1789763173.054676:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763173.054685:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504309888:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763173.054695:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504310528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763173.054726:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504310016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763173.054759:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763173.054775:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763173/real 1789763173] req@ffff9018c66a5500 x1876699504310016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763173.054790:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504310016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763173.054801:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763173.054805:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763173.054835:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504310016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763173.054843:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504310144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763173.054867:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763173.054879:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763173/real 1789763173] req@ffff9018c66a6680 x1876699504310144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763173.054903:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504310144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763173.054912:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763173.054916:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763173.054919:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504310144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763173.054927:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504310272:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763173.054946:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763173.054957:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763173/real 1789763173] req@ffff9018c66a4700 x1876699504310272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763173.054971:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504310272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763173.054980:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763173.054983:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763173.054987:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504310272:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763173.054993:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504310400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763173.055010:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763173.055021:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763173/real 1789763173] req@ffff9018c66a6300 x1876699504310400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763173.055078:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504310400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763173.055098:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763173.055101:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763173.055106:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504310400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763173.055112:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504310528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763173.055138:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763173.055151:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763173/real 1789763173] req@ffff9018c66a5180 x1876699504310528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763173.055165:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504310528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763173.055174:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763173.055178:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763173.055181:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504310528:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763173.694321:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763174.718306:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763175.742341:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763176.766341:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763177.790340:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763178.174184:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763178.174206:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763178.174215:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763178.174237:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3104 00000100:00080000:1.0:1789763178.174260:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763178.174276:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763178.174340:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504310656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763178.174404:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763178.174410:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763178.174413:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763178.174420:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3104 00000100:00080000:1.0:1789763178.174427:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763178.174434:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763178.174450:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763178.174470:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504310784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763178.174486:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763178.174491:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763178.174494:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763178.174500:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3104 00000100:00080000:1.0:1789763178.174505:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763178.174510:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763178.174517:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763178.174536:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504310912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763178.174554:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504310656:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763178.174558:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763178.174565:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763178.174568:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763178.174577:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3104 00000100:00080000:1.0:1789763178.174586:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763178.174596:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763178.174609:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763178.174634:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504311040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763178.174657:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763178.174665:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763178.174669:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763178.174679:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3104 00000100:00080000:1.0:1789763178.174688:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763178.174697:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763178.174707:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763178.174733:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504311168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763178.174752:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763178.174759:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763178.174799:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763178.174808:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3104 00000100:00080000:1.0:1789763178.174817:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763178.174827:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763178.174839:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763178.174862:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504311296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763178.175235:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763178.175290:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763178/real 1789763178] req@ffff9018c66a7800 x1876699504310656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763178.175316:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504310656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763178.175331:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763178.175336:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763178.175342:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504310656:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763178.175356:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504310784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763178.175382:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763178.175398:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763178/real 1789763178] req@ffff9018c66a7b80 x1876699504310784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763178.175413:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504310784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763178.175423:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763178.175427:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763178.175432:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504310784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763178.175465:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504310912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763178.175486:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763178.175513:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763178/real 1789763178] req@ffff9018c66a4000 x1876699504310912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763178.175530:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504310912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763178.175540:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763178.175544:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763178.175549:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504310912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763178.175556:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504311040:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763178.175577:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763178.175590:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763178/real 1789763178] req@ffff9018c66a5f80 x1876699504311040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763178.175604:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504311040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763178.175614:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763178.175617:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763178.175621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504311040:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763178.175629:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504311168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763178.175647:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763178.175659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763178/real 1789763178] req@ffff9018c66a6d80 x1876699504311168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763178.175673:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504311168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763178.175683:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763178.175686:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763178.175690:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504311168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763178.175696:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504311296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763178.175715:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763178.175726:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763178/real 1789763178] req@ffff9018c66a5c00 x1876699504311296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763178.175740:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504311296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763178.175749:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763178.175753:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763178.175756:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504311296:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763178.814259:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763179.838237:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763180.862295:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763181.886304:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763182.910282:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763183.294235:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763183.294252:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763183.294259:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763183.294275:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3109 00000100:00080000:2.0:1789763183.294293:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763183.294305:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763183.294384:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504311424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763183.294443:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763183.294449:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763183.294451:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763183.294458:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3109 00000100:00080000:2.0:1789763183.294464:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763183.294471:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763183.294487:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763183.294515:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504311552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763183.294532:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763183.294539:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763183.294542:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763183.294549:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3109 00000100:00080000:2.0:1789763183.294558:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763183.294562:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504311424:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763183.294571:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763183.294592:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763183.294611:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504311680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763183.294636:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763183.294641:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763183.294645:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763183.294653:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3109 00000100:00080000:2.0:1789763183.294671:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763183.294681:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789763183.294683:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789763183.294707:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763183.294727:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504311808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763183.294773:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763183.294781:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1789763183.294789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763183/real 1789763183] req@ffff9018d0f25500 x1876699504311424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763183.294790:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763183.294817:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3109 00000100:00080000:2.0:1789763183.294837:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763183.294844:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504311424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763183.294853:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763183.294885:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789763183.294886:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763183.294900:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763183.294907:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504311424:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763183.294934:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504311936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763183.294952:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504311552:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763183.294986:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763183.294993:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763183.294997:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763183.295006:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3109 00000800:00000100:3.0:1789763183.295016:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763183.295022:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763183.295081:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763183.295093:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763183.295093:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763183/real 1789763183] req@ffff9018d0f27480 x1876699504311552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763183.295127:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504312064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763183.295130:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504311552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763183.295227:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763183.295235:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763183.295243:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504311552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763183.295255:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504311680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763183.295286:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763183.295300:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763183/real 1789763183] req@ffff9018d0f26680 x1876699504311680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763183.295314:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504311680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763183.295324:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763183.295328:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763183.295332:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504311680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763183.295340:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504311808:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763183.295360:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763183.295372:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763183/real 1789763183] req@ffff9018d0f24700 x1876699504311808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763183.295386:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504311808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763183.295396:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763183.295400:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763183.295403:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504311808:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763183.295426:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504311936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763183.295445:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763183.295458:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763183/real 1789763183] req@ffff9018d0f24000 x1876699504311936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763183.295472:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504311936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763183.295481:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763183.295485:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763183.295488:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504311936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763183.295495:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504312064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763183.295513:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763183.295524:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763183/real 1789763183] req@ffff9018d0f24e00 x1876699504312064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763183.295539:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504312064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763183.295548:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763183.295552:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763183.295555:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504312064:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763183.934319:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763184.958333:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763185.982235:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763187.006342:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763188.030321:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763188.414258:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763188.414278:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763188.414287:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763188.414308:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3114 00000100:00080000:1.0:1789763188.414330:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763188.414356:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763188.414432:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504312192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763188.414481:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763188.414497:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763188.414500:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763188.414507:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3114 00000100:00080000:1.0:1789763188.414513:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763188.414520:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763188.414536:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763188.414559:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504312320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763188.414568:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504312192:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763188.414587:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763188.414592:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763188.414595:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763188.414600:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3114 00000100:00080000:1.0:1789763188.414604:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763188.414609:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763188.414616:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763188.414633:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504312448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763188.414657:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763188.414662:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763188.414664:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763188.414669:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3114 00000100:00080000:1.0:1789763188.414673:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763188.414678:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763188.414684:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789763188.414726:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763188.414732:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504312576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763188.414759:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763188.414764:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763188.414768:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763188.414774:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3114 00000100:00080000:1.0:1789763188.414781:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763188.414786:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763188.414793:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763188.414811:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504312704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763188.414823:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763188.414828:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763188.414835:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763188.414836:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763188/real 1789763188] req@ffff9018c66a5c00 x1876699504312192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763188.414846:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3114 00000100:00080000:1.0:1789763188.414856:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763188.414867:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763188.414874:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763188.414887:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504312192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763188.414897:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504312832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763188.414926:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763188.414936:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763188.414944:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504312192:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763188.414985:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504312320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763188.415019:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763188.415074:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763188/real 1789763188] req@ffff9018c66a6d80 x1876699504312320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763188.415093:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504312320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763188.415123:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763188.415127:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763188.415132:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504312320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763188.415139:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504312448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763188.415164:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763188.415205:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763188/real 1789763188] req@ffff9018c66a5f80 x1876699504312448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763188.415221:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504312448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763188.415232:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763188.415236:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763188.415240:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504312448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763188.415247:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504312576:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763188.415271:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763188.415290:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763188/real 1789763188] req@ffff9018c66a4000 x1876699504312576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763188.415314:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504312576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763188.415331:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763188.415349:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763188.415357:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504312576:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763188.415370:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504312704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763188.415408:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763188.415426:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763188/real 1789763188] req@ffff9018c66a7b80 x1876699504312704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763188.415442:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504312704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763188.415452:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763188.415456:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763188.415461:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504312704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763188.415468:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504312832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763188.415498:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763188.415510:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763188/real 1789763188] req@ffff9018c66a7800 x1876699504312832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763188.415524:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504312832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763188.415534:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763188.415537:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763188.415541:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504312832:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763189.054313:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763190.078333:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763191.102320:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763192.126294:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763193.150221:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763193.534203:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763193.534217:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763193.534223:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763193.534236:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3120 00000100:00080000:0.0:1789763193.534250:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763193.534261:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763193.534325:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504312960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763193.534359:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763193.534364:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763193.534367:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763193.534372:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3120 00000100:00080000:0.0:1789763193.534376:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763193.534381:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763193.534394:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763193.534411:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504313088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763193.534425:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763193.534429:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763193.534432:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763193.534437:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3120 00000100:00080000:0.0:1789763193.534441:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763193.534449:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763193.534453:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504312960:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789763193.534458:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763193.534479:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504313216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763193.534491:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763193.534495:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763193.534498:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763193.534502:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3120 00000100:00080000:0.0:1789763193.534506:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763193.534511:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763193.534517:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763193.534531:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504313344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763193.534543:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763193.534547:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763193.534550:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763193.534555:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3120 00000100:00080000:0.0:1789763193.534559:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763193.534569:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789763193.534574:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789763193.534578:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763193.534595:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504313472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763193.534608:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763193.534612:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763193.534615:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763193.534620:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3120 00000100:00080000:0.0:1789763193.534624:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763193.534628:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763193.534634:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763193.534654:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504313600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763193.534690:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763193/real 1789763193] req@ffff9018c8b99f80 x1876699504312960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763193.534735:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504312960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763193.534751:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763193.534758:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763193.534764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504312960:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763193.534775:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504313088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763193.534808:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763193.534825:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763193/real 1789763193] req@ffff9018c8b98700 x1876699504313088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763193.534841:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504313088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763193.534852:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763193.534856:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763193.534860:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504313088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763193.534893:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504313216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763193.534931:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763193.534954:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763193/real 1789763193] req@ffff9018c8b9bb80 x1876699504313216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763193.534979:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504313216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763193.535007:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763193.535015:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763193.535086:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504313216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763193.535103:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504313344:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763193.535150:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763193.535178:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763193/real 1789763193] req@ffff9018c8b9b100 x1876699504313344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763193.535204:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504313344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763193.535222:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763193.535227:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763193.535232:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504313344:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763193.535240:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504313472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763193.535270:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763193.535285:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763193/real 1789763193] req@ffff9018c8b9b480 x1876699504313472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763193.535300:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504313472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763193.535309:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763193.535313:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763193.535318:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504313472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763193.535325:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504313600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763193.535344:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763193.535355:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763193/real 1789763193] req@ffff9018c8b98a80 x1876699504313600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763193.535372:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504313600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763193.535383:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763193.535389:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763193.535393:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504313600:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763194.174300:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763195.198309:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763196.222270:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763197.246365:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763198.270305:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763198.654223:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763198.654238:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763198.654246:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763198.654263:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3125 00000100:00080000:2.0:1789763198.654289:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763198.654301:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763198.654366:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504313728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763198.654410:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763198.654416:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763198.654420:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763198.654426:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3125 00000100:00080000:2.0:1789763198.654431:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763198.654436:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763198.654454:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763198.654463:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504313728:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763198.654478:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504313856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763198.654494:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763198.654500:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763198.654502:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763198.654507:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3125 00000100:00080000:2.0:1789763198.654511:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763198.654515:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763198.654522:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763198.654538:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504313984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763198.654551:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763198.654556:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763198.654559:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763198.654564:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3125 00000100:00080000:2.0:1789763198.654570:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763198.654580:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789763198.654585:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789763198.654590:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763198.654607:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504314112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763198.654648:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763198.654653:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763198.654655:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763198.654660:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3125 00000100:00080000:2.0:1789763198.654666:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763198.654671:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763198.654681:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763198.654700:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763198/real 1789763198] req@ffff9018d0f27800 x1876699504313728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763198.654704:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504314240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763198.654770:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763198.654776:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763198.654779:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763198.654785:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3125 00000100:00080000:2.0:1789763198.654790:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763198.654796:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504313728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763198.654803:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763198.654824:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789763198.654830:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763198.654840:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763198.654849:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504313728:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763198.654857:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504314368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763198.654895:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504313856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763198.654937:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763198.654957:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763198/real 1789763198] req@ffff9018d0f25c00 x1876699504313856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763198.654974:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504313856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763198.654985:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763198.654990:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763198.654994:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504313856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763198.655001:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504313984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763198.655021:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763198.655033:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763198/real 1789763198] req@ffff9018d0f26a00 x1876699504313984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763198.655091:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504313984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763198.655102:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763198.655106:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763198.655110:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504313984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763198.655117:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504314112:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763198.655142:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763198.655156:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763198/real 1789763198] req@ffff9018d0f24a80 x1876699504314112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763198.655170:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504314112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763198.655181:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763198.655185:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763198.655189:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504314112:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763198.655196:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504314240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763198.655214:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763198.655225:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763198/real 1789763198] req@ffff9018d0f24380 x1876699504314240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763198.655239:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504314240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763198.655249:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763198.655252:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763198.655256:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504314240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763198.655277:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504314368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763198.655296:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763198.655308:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763198/real 1789763198] req@ffff9018d0f25180 x1876699504314368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763198.655321:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504314368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763198.655330:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763198.655334:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763198.655339:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504314368:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763199.294252:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763200.318336:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763201.342252:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763202.366235:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763203.390307:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763203.774143:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763203.774171:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763203.774178:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763203.774194:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3130 00000100:00080000:2.0:1789763203.774211:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763203.774224:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763203.774299:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504314496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763203.774355:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763203.774361:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763203.774364:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763203.774371:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3130 00000100:00080000:2.0:1789763203.774377:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763203.774385:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763203.774401:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763203.774425:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504314624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763203.774441:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763203.774451:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789763203.774453:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504314496:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763203.774456:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763203.774478:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3130 00000100:00080000:2.0:1789763203.774484:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763203.774489:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763203.774496:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763203.774517:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504314752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763203.774530:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763203.774535:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763203.774538:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763203.774543:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3130 00000100:00080000:2.0:1789763203.774561:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789763203.774574:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763203.774581:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763203.774644:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763203.774679:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504314880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763203.774695:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763203.774700:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763203.774704:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763203.774709:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3130 00000100:00080000:2.0:1789763203.774716:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763203.774721:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763203/real 1789763203] req@ffff9018d0f26300 x1876699504314496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763203.774729:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763203.774749:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763203.774777:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504315008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763203.774785:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504314496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763203.774822:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763203.774826:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763203.774829:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763203.774845:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3130 00000100:00080000:2.0:1789763203.774860:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763203.774875:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789763203.774879:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789763203.774883:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789763203.774892:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763203.774917:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504314496:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763203.774926:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504315136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763203.774989:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504314624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763203.775092:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763203.775117:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763203/real 1789763203] req@ffff9018d0f27100 x1876699504314624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763203.775154:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504314624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763203.775165:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763203.775170:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763203.775174:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504314624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763203.775183:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504314752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763203.775209:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763203.775223:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763203/real 1789763203] req@ffff9018d0f25f80 x1876699504314752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763203.775238:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504314752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763203.775247:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763203.775251:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763203.775255:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504314752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763203.775262:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504314880:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763203.775281:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763203.775291:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763203/real 1789763203] req@ffff9018d0f26d80 x1876699504314880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763203.775305:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504314880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763203.775314:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763203.775318:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763203.775323:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504314880:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763203.775335:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504315008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763203.775369:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763203.775391:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763203/real 1789763203] req@ffff9018d0f25180 x1876699504315008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763203.775416:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504315008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763203.775429:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763203.775433:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763203.775437:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504315008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763203.775445:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504315136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763203.775471:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763203.775485:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763203/real 1789763203] req@ffff9018d0f24380 x1876699504315136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763203.775499:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504315136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763203.775508:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763203.775512:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763203.775515:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504315136:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763204.414256:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763205.438362:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763206.462287:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763207.486252:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763208.510354:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763208.894181:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763208.894194:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763208.894198:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763208.894220:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3135 00000100:00080000:1.0:1789763208.894233:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763208.894241:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763208.894284:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504315264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763208.894316:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763208.894321:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763208.894324:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763208.894329:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3135 00000100:00080000:1.0:1789763208.894333:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763208.894337:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763208.894349:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763208.894364:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504315392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763208.894410:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763208.894417:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789763208.894417:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504315264:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763208.894424:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763208.894440:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3135 00000100:00080000:1.0:1789763208.894445:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763208.894449:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763208.894457:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763208.894473:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504315520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763208.894487:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763208.894492:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763208.894494:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763208.894498:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3135 00000100:00080000:1.0:1789763208.894504:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763208.894509:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763208.894515:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789763208.894523:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763208.894537:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504315648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763208.894578:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763208.894583:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763208.894586:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763208.894608:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3135 00000100:00080000:1.0:1789763208.894615:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763208.894621:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763208.894628:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763208.894632:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763208/real 1789763208] req@ffff9018c66a5180 x1876699504315264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763208.894648:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504315776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763208.894680:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763208.894688:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504315264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763208.894689:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763208.894695:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763208.894710:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3135 00000100:00080000:1.0:1789763208.894720:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763208.894728:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763208.894729:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763208.894744:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789763208.894745:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763208.894774:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504315264:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763208.894782:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504315904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763208.894816:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504315392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763208.894855:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763208.894878:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763208/real 1789763208] req@ffff9018c66a6300 x1876699504315392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763208.894904:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504315392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763208.894921:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763208.894928:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763208.894934:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504315392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763208.894946:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504315520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763208.894974:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763208.894989:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763208/real 1789763208] req@ffff9018c66a4700 x1876699504315520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763208.895002:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504315520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763208.895012:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763208.895016:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763208.895019:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504315520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763208.895026:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504315648:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763208.895043:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763208.895055:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763208/real 1789763208] req@ffff9018c66a6680 x1876699504315648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763208.895068:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504315648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763208.895077:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763208.895116:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763208.895121:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504315648:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763208.895128:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504315776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763208.895165:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763208.895180:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763208/real 1789763208] req@ffff9018c66a5500 x1876699504315776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763208.895194:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504315776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763208.895203:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763208.895207:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763208.895210:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504315776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763208.895217:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504315904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763208.895242:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763208.895262:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763208/real 1789763208] req@ffff9018c66a4e00 x1876699504315904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763208.895284:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504315904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763208.895301:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763208.895308:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763208.895314:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504315904:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763209.534285:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763210.558247:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763211.582265:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763212.606280:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763213.630301:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763214.014226:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763214.014244:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763214.014251:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763214.014270:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3140 00000100:00080000:2.0:1789763214.014286:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763214.014297:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763214.014354:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504316032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763214.014406:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763214.014411:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763214.014413:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763214.014418:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3140 00000100:00080000:2.0:1789763214.014423:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763214.014429:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763214.014442:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763214.014465:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504316160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763214.014482:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763214.014487:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763214.014490:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763214.014504:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504316032:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763214.014529:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3140 00000100:00080000:2.0:1789763214.014565:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763214.014581:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763214.014592:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763214.014616:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504316288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763214.014630:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763214.014637:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763214.014640:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763214.014651:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763214.014655:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3140 00000100:00080000:2.0:1789763214.014665:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763214.014676:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763214.014688:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763214.014710:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504316416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763214.014724:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763214.014734:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789763214.014736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763214/real 1789763214] req@ffff9018d0f24a80 x1876699504316032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763214.014737:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763214.014751:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3140 00000100:00080000:2.0:1789763214.014761:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763214.014774:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763214.014796:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763214.014820:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504316544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763214.014834:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504316032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763214.014838:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763214.014874:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763214.014884:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763214.014890:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763214.014894:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3140 00000100:00080000:0.0:1789763214.014905:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763214.014924:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504316032:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763214.014929:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763214.014949:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763214.014956:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763214.014969:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504316160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763214.014980:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504316672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763214.015042:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763214.015123:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763214/real 1789763214] req@ffff9018d0f26a00 x1876699504316160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763214.015145:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504316160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763214.015164:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763214.015169:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763214.015173:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504316160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763214.015181:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504316288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763214.015208:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763214.015220:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763214/real 1789763214] req@ffff9018d0f25c00 x1876699504316288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763214.015236:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504316288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763214.015252:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763214.015260:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763214.015266:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504316288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763214.015278:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504316416:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763214.015340:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763214.015367:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763214/real 1789763214] req@ffff9018d0f27800 x1876699504316416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763214.015387:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504316416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763214.015398:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763214.015403:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763214.015409:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504316416:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763214.015419:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504316544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763214.015455:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763214.015474:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763214/real 1789763214] req@ffff9018d0f24e00 x1876699504316544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763214.015490:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504316544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763214.015502:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763214.015506:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763214.015547:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504316544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763214.015624:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504316672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763214.015666:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763214.015689:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763214/real 1789763214] req@ffff9018d0f24000 x1876699504316672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763214.015704:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504316672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763214.015715:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763214.015723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763214.015729:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504316672:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763214.654379:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763215.678350:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763216.702284:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763217.726256:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763218.750234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763219.134207:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763219.134221:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763219.134226:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763219.134240:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3145 00000100:00080000:0.0:1789763219.134253:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763219.134263:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763219.134311:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504316800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763219.134349:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763219.134354:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763219.134357:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763219.134362:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3145 00000100:00080000:0.0:1789763219.134367:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763219.134373:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763219.134387:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763219.134412:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504316928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763219.134414:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504316800:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763219.134443:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763219.134447:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763219.134450:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763219.134455:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3145 00000100:00080000:0.0:1789763219.134460:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763219.134465:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763219.134495:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763219.134517:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504317056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763219.134539:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763219.134547:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1789763219.134549:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763219.134551:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763219.134561:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3145 00000100:00080000:0.0:1789763219.134569:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763219.134575:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763219.134583:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763219.134600:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504317184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763219.134613:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763219.134616:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763219/real 1789763219] req@ffff9018c8b99880 x1876699504316800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763219.134618:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763219.134626:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763219.134639:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3145 00000100:00080000:0.0:1789763219.134651:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763219.134657:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763219.134664:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763219.134668:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504316800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763219.134683:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504317312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763219.134694:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763219.134701:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763219.134707:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789763219.134710:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763219.134717:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763219.134721:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504316800:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763219.134728:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3145 00000100:00080000:0.0:1789763219.134736:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763219.134741:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763219.134747:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763219.134761:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504317440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763219.134770:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504316928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763219.134831:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763219.134859:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763219/real 1789763219] req@ffff9018c8b9b800 x1876699504316928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763219.134904:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504316928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763219.134916:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763219.134921:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763219.134926:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504316928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763219.134935:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504317056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763219.134966:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763219.134981:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763219/real 1789763219] req@ffff9018c8b98000 x1876699504317056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763219.134997:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504317056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763219.135006:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763219.135011:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763219.135015:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504317056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763219.135022:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504317184:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763219.135079:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763219.135095:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763219/real 1789763219] req@ffff9018c8b9aa00 x1876699504317184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763219.135111:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504317184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763219.135121:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763219.135125:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763219.135129:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504317184:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763219.135136:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504317312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763219.135156:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763219.135167:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763219/real 1789763219] req@ffff9018c8b98e00 x1876699504317312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763219.135182:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504317312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763219.135192:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763219.135195:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763219.135199:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504317312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763219.135222:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504317440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763219.135242:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763219.135255:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763219/real 1789763219] req@ffff9018c8b9a680 x1876699504317440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763219.135282:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504317440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763219.135293:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763219.135297:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763219.135301:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504317440:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763219.774280:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763220.798227:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763221.822205:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763222.846216:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763223.870232:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763224.254106:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763224.254118:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763224.254123:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763224.254136:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3150 00000100:00080000:0.0:1789763224.254150:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763224.254160:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763224.254228:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504317568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763224.254277:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763224.254285:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763224.254289:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763224.254297:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3150 00000100:00080000:0.0:1789763224.254307:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763224.254317:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763224.254332:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763224.254357:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504317696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763224.254362:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504317568:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763224.254396:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763224.254405:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763224.254409:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763224.254419:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3150 00000100:00080000:0.0:1789763224.254430:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763224.254462:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763224.254471:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763224.254509:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504317824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763224.254520:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763224.254524:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763224.254526:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763224.254530:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3150 00000100:00080000:0.0:1789763224.254535:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763224.254540:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763224.254546:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763224.254562:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504317696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763224.254563:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504317952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763224.254584:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763224.254588:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763224.254590:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763224.254594:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3150 00000100:00080000:0.0:1789763224.254600:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763224.254604:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763224.254613:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763224.254639:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504317824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763224.254650:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504318080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763224.254665:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763224.254670:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763224.254673:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763224.254679:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3150 00000100:00080000:0.0:1789763224.254684:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763224.254689:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763224.254696:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763224.254710:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504317952:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763224.254720:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504318208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763224.254781:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504318080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763224.254821:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504318208:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763224.894272:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763225.918299:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763226.942259:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763227.326348:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789763227.326402:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763227.326454:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763227.326515:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763227.326533:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763227.326555:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763227.326577:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763227.326602:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:3.0:1789763227.326606:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763224/real 1789763227] req@ffff9018c8b9ad80 x1876699504317952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763227.326660:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504317952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763227.326683:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763227.326692:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763227.326701:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504317952:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763227.326716:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763224/real 1789763227] req@ffff9018c8b9a300 x1876699504318080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763227.326742:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504318080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763227.326758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763227.326765:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763227.326771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504318080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763227.326783:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763224/real 1789763227] req@ffff9018c8b98380 x1876699504317696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763227.326803:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504317696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763227.326817:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763227.326825:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763227.326831:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504317696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763227.326843:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763224/real 1789763227] req@ffff9018c8b99500 x1876699504317824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763227.326864:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504317824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763227.326879:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763227.326886:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763227.326893:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504317824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763227.326905:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763224/real 1789763227] req@ffff9018c8b99c00 x1876699504317568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763227.326930:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504317568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763227.326946:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763227.326952:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763227.326958:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504317568:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763227.327006:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763224/real 1789763227] req@ffff9018c8b9a680 x1876699504318208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763227.327030:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504318208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763227.327088:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763227.327096:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763227.327102:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504318208:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763227.966176:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763228.990237:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763229.374187:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763229.374200:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763229.374206:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763229.374219:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3155 00000100:00080000:1.0:1789763229.374233:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763229.374241:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763229.374296:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504318336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763229.374335:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763229.374341:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763229.374344:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763229.374349:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3155 00000100:00080000:1.0:1789763229.374354:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763229.374358:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763229.374371:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763229.374387:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504318464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763229.374425:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763229.374430:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763229.374443:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763229.374449:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3155 00000100:00080000:1.0:1789763229.374455:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763229.374461:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763229.374480:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763229.374480:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504318336:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763229.374508:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504318592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763229.374523:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763229.374528:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763229.374530:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763229.374536:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3155 00000100:00080000:1.0:1789763229.374542:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763229.374550:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763229.374561:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763229.374597:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504318720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763229.374601:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763229.374629:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763229.374636:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763229.374640:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763229.374649:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3155 00000100:00080000:1.0:1789763229.374658:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763229.374668:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763229.374680:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763229.374683:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763229/real 1789763229] req@ffff9018c66a5880 x1876699504318336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763229.374711:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504318848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763229.374728:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504318336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763229.374743:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763229.374762:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763229.374770:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763229.374775:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763229.374783:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3155 00000100:00080000:3.0:1789763229.374793:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763229.374806:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504318336:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763229.374808:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763229.374825:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763229.374836:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504318464:192.168.203.158@tcp:38:kworker.0 00000008:00100000:1.0:1789763229.374840:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763229.374878:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504318976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763229.374904:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763229.374941:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763229/real 1789763229] req@ffff9018c66a7480 x1876699504318464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763229.374972:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504318464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763229.374989:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763229.374997:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763229.375003:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504318464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763229.375045:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504318592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763229.375170:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763229.375196:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763229/real 1789763229] req@ffff9018c66a4a80 x1876699504318592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763229.375220:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504318592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763229.375235:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763229.375241:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763229.375247:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504318592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763229.375899:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504318720:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763229.375955:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763229.375985:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763229/real 1789763229] req@ffff9018c66a7100 x1876699504318720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763229.376013:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504318720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763229.376031:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763229.376083:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763229.376091:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504318720:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763229.376104:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504318848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763229.376168:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763229.376188:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763229/real 1789763229] req@ffff9018c66a4380 x1876699504318848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763229.376214:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504318848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763229.376225:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763229.376229:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763229.376233:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504318848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763229.376240:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504318976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763229.376263:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763229.376285:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763229/real 1789763229] req@ffff9018c66a6a00 x1876699504318976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763229.376299:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504318976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763229.376317:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763229.376321:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763229.376325:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504318976:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763230.014232:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763231.038301:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763232.062264:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763233.086267:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763234.110259:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763234.494159:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763234.494167:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763234.494171:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763234.494179:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3161 00000100:00080000:2.0:1789763234.494189:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763234.494195:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763234.494228:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504319104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763234.494254:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763234.494257:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763234.494259:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763234.494262:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3161 00000100:00080000:2.0:1789763234.494265:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763234.494268:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763234.494277:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763234.494287:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504319232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763234.494295:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763234.494298:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763234.494300:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763234.494303:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3161 00000100:00080000:2.0:1789763234.494305:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763234.494310:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504319104:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763234.494311:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763234.494320:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763234.494330:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504319360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763234.494338:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763234.494340:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763234.494342:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763234.494345:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3161 00000100:00080000:2.0:1789763234.494383:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763234.494388:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789763234.494393:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789763234.494398:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763234.494413:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504319488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763234.494421:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763234.494424:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763234.494426:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763234.494430:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3161 00000100:00080000:2.0:1789763234.494433:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763234.494436:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763234/real 1789763234] req@ffff9018d0f24700 x1876699504319104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763234.494440:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763234.494448:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763234.494463:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504319616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763234.494468:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504319104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763234.494509:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763234.494523:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763234.494529:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763234.494543:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3161 00000100:00080000:3.0:1789763234.494547:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763234.494549:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763234.494563:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763234.494564:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763234.494577:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763234.494580:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504319104:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763234.494597:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504319744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763234.494599:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504319232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763234.494652:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763234.494674:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763234/real 1789763234] req@ffff9018d0f26680 x1876699504319232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763234.494691:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504319232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763234.494702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763234.494706:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763234.494710:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504319232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763234.494738:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504319360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763234.494759:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763234.494771:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763234/real 1789763234] req@ffff9018d0f27480 x1876699504319360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763234.494785:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504319360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763234.494811:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763234.494813:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763234.494816:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504319360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763234.494821:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504319488:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763234.494834:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763234.494841:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763234/real 1789763234] req@ffff9018d0f25500 x1876699504319488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763234.494850:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504319488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763234.494856:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763234.494858:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763234.494860:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504319488:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763234.494865:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504319616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763234.494880:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763234.494891:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763234/real 1789763234] req@ffff9018d0f24000 x1876699504319616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763234.494901:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504319616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763234.494908:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763234.494913:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763234.494916:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504319616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763234.494922:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504319744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763234.494935:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763234.494942:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763234/real 1789763234] req@ffff9018d0f24e00 x1876699504319744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763234.494951:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504319744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763234.494965:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763234.494970:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763234.494975:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504319744:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763235.134294:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763236.158245:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763237.182271:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763238.206270:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763239.230347:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763239.614202:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763239.614224:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763239.614234:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763239.614253:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3166 00000100:00080000:2.0:1789763239.614272:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763239.614283:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763239.614399:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504319872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763239.614443:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763239.614449:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763239.614452:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763239.614458:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3166 00000100:00080000:2.0:1789763239.614463:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763239.614470:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763239.614485:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763239.614504:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504320000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763239.614519:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763239.614524:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763239.614527:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763239.614531:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3166 00000100:00080000:2.0:1789763239.614535:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763239.614540:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763239.614548:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763239.614552:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504319872:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763239.614566:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504320128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763239.614581:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763239.614586:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763239.614588:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763239.614592:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3166 00000100:00080000:2.0:1789763239.614596:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763239.614600:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763239.614606:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763239.614620:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504320256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763239.614632:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763239.614638:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763239.614641:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763239.614646:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3166 00000100:00080000:2.0:1789763239.614650:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763239.614655:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763239.614660:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763239.614673:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504320384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763239.614684:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763239.614688:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763239.614690:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763239.614695:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3166 00000100:00080000:2.0:1789763239.614700:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763239.614704:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763239.614710:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763239.614716:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763239.614728:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504320512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763239.614806:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763239/real 1789763239] req@ffff9018d0f27800 x1876699504319872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763239.614838:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504319872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763239.614854:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763239.614860:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763239.614866:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504319872:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763239.614877:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504320000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763239.614912:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763239.614938:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763239/real 1789763239] req@ffff9018d0f25c00 x1876699504320000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763239.614964:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504320000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763239.615023:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763239.615031:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763239.615111:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504320000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763239.615158:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504320128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763239.615201:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763239.615222:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763239/real 1789763239] req@ffff9018d0f26a00 x1876699504320128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763239.615241:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504320128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763239.615253:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763239.615259:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763239.615264:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504320128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763239.615272:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504320256:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763239.615293:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763239.615305:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763239/real 1789763239] req@ffff9018d0f24a80 x1876699504320256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763239.615368:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504320256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763239.615380:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763239.615385:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763239.615391:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504320256:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763239.615398:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504320384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763239.615423:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763239.615436:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763239/real 1789763239] req@ffff9018d0f24380 x1876699504320384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763239.615451:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504320384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763239.615461:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763239.615465:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763239.615469:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504320384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763239.615476:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504320512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763239.615495:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763239.615506:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763239/real 1789763239] req@ffff9018d0f25180 x1876699504320512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763239.615520:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504320512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763239.615530:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763239.615533:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763239.615536:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504320512:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763240.254353:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763241.278346:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763242.302281:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763243.326266:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763244.350267:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763244.734196:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763244.734212:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763244.734220:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763244.734236:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3171 00000100:00080000:1.0:1789763244.734254:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763244.734265:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763244.734363:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504320640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763244.734408:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763244.734417:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763244.734421:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763244.734429:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3171 00000100:00080000:1.0:1789763244.734436:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763244.734444:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763244.734463:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763244.734489:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504320768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763244.734505:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504320640:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763244.734510:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763244.734517:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763244.734520:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763244.734526:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3171 00000100:00080000:1.0:1789763244.734533:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763244.734540:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763244.734549:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763244.734566:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504320896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763244.734592:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763244.734598:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763244.734601:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763244.734608:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3171 00000100:00080000:1.0:1789763244.734614:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789763244.734620:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763244.734621:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763244.734629:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763244.734646:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504321024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763244.734663:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763244.734669:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763244.734672:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763244.734677:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3171 00000100:00080000:1.0:1789763244.734683:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763244.734690:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763244.734696:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763244/real 1789763244] req@ffff9018c66a6a00 x1876699504320640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789763244.734716:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763244.734732:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504320640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763244.734735:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504321152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763244.734755:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763244.734755:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763244.734762:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763244.734763:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789763244.734766:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763244.734772:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504320640:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763244.734773:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3171 00000100:00080000:1.0:1789763244.734781:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763244.734787:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763244.734794:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763244.734809:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504321280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763244.734818:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504320768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763244.734864:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763244.734892:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763244/real 1789763244] req@ffff9018c66a4380 x1876699504320768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763244.734920:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504320768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763244.734939:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763244.734945:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763244.734951:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504320768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763244.734964:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504320896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763244.735009:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763244.735083:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763244/real 1789763244] req@ffff9018c66a7100 x1876699504320896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763244.735112:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504320896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763244.735123:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763244.735127:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763244.735132:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504320896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763244.735139:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504321024:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763244.735171:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763244.735185:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763244/real 1789763244] req@ffff9018c66a4a80 x1876699504321024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763244.735201:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504321024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763244.735211:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763244.735215:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763244.735219:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504321024:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763244.735225:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504321152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763244.735246:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763244.735258:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763244/real 1789763244] req@ffff9018c66a7480 x1876699504321152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763244.735272:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504321152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763244.735331:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763244.735336:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763244.735340:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504321152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763244.735368:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504321280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763244.735397:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763244.735412:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763244/real 1789763244] req@ffff9018c66a5880 x1876699504321280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763244.735427:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504321280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763244.735437:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763244.735441:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763244.735445:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504321280:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763245.374272:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763246.398353:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763247.422234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763248.446316:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763249.470315:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763249.854216:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763249.854232:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763249.854239:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763249.854297:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3176 00000100:00080000:1.0:1789763249.854315:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763249.854327:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763249.854403:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504321408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763249.854450:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763249.854456:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763249.854459:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763249.854466:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3176 00000100:00080000:1.0:1789763249.854471:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763249.854477:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763249.854494:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763249.854519:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504321536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763249.854539:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763249.854549:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789763249.854554:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504321408:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763249.854555:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763249.854573:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3176 00000100:00080000:1.0:1789763249.854581:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763249.854587:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763249.854595:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763249.854613:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504321664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763249.854627:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763249.854633:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763249.854635:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763249.854640:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3176 00000100:00080000:1.0:1789763249.854644:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763249.854658:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763249.854665:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763249.854679:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504321792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763249.854694:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763249.854714:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763249.854719:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763249.854722:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763249.854732:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3176 00000100:00080000:1.0:1789763249.854738:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763249.854743:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763249.854752:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763249.854761:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763249/real 1789763249] req@ffff9018c66a4e00 x1876699504321408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763249.854773:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504321920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763249.854842:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763249.854858:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763249.854861:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763249.854872:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504321408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763249.854875:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3176 00000100:00080000:1.0:1789763249.854888:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763249.854921:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763249.854923:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789763249.854945:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:1.0:1789763249.854946:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763249.854963:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504321408:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763249.854986:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504322048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763249.855025:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504321536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763249.855127:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763249.855157:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763249/real 1789763249] req@ffff9018c66a5500 x1876699504321536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763249.855177:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504321536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763249.855191:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763249.855214:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763249.855220:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504321536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763249.855231:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504321664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763249.855352:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763249.855374:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763249/real 1789763249] req@ffff9018c66a6680 x1876699504321664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763249.855398:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504321664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763249.855408:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763249.855412:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763249.855419:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504321664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763249.855429:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504321792:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763249.855458:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763249.855476:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763249/real 1789763249] req@ffff9018c66a4700 x1876699504321792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763249.855496:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504321792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763249.855534:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763249.855541:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763249.855548:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504321792:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763249.855562:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504321920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763249.855598:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763249.855614:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763249/real 1789763249] req@ffff9018c66a6300 x1876699504321920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763249.855628:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504321920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763249.855647:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763249.855651:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763249.855655:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504321920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763249.855681:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504322048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763249.855701:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763249.855711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763249/real 1789763249] req@ffff9018c66a5180 x1876699504322048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763249.855723:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504322048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763249.855731:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763249.855734:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763249.855738:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504322048:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763250.494272:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763251.518303:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763252.542339:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763253.566313:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763254.590303:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763254.974182:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763254.974198:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763254.974205:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763254.974246:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3181 00000100:00080000:1.0:1789763254.974263:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763254.974275:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763254.974349:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504322176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763254.974406:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763254.974414:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763254.974417:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763254.974425:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3181 00000100:00080000:1.0:1789763254.974433:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763254.974442:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763254.974460:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763254.974486:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504322304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763254.974490:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504322176:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763254.974533:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763254.974539:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763254.974542:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763254.974549:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3181 00000100:00080000:1.0:1789763254.974553:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763254.974558:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763254.974566:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763254.974580:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504322432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763254.974592:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763254.974598:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763254.974601:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763254.974606:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3181 00000100:00080000:1.0:1789763254.974611:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763254.974616:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763254.974623:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763254.974643:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504322560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763254.974650:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763254.974678:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763254.974683:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763254.974685:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763254.974690:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3181 00000100:00080000:1.0:1789763254.974696:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763254.974701:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763254.974707:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763254.974724:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504322688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763254.974739:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763254.974745:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763254.974748:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763254.974760:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3181 00000100:00100000:2.0:1789763254.974764:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763254/real 1789763254] req@ffff9018c66a7800 x1876699504322176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763254.974768:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763254.974786:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763254.974809:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763254.974850:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504322816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763254.974853:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504322176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763254.974936:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763254.974945:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763254.974953:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504322176:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763254.974994:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504322304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763254.975030:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763254.975084:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763254/real 1789763254] req@ffff9018c66a7b80 x1876699504322304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763254.975101:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504322304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763254.975111:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763254.975116:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763254.975120:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504322304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763254.975128:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504322432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763254.975153:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763254.975165:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763254/real 1789763254] req@ffff9018c66a4000 x1876699504322432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763254.975178:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504322432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763254.975188:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763254.975192:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763254.975195:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504322432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763254.975203:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504322560:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763254.975268:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763254.975291:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763254/real 1789763254] req@ffff9018c66a5f80 x1876699504322560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763254.975316:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504322560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763254.975330:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763254.975334:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763254.975338:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504322560:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763254.975347:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504322688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763254.975375:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763254.975388:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763254/real 1789763254] req@ffff9018c66a6d80 x1876699504322688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763254.975402:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504322688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763254.975412:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763254.975415:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763254.975419:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504322688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763254.975426:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504322816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763254.975446:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763254.975457:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763254/real 1789763254] req@ffff9018c66a5c00 x1876699504322816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763254.975471:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504322816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763254.975481:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763254.975484:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763254.975488:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504322816:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763255.614272:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763256.638165:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763257.662277:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763258.686167:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763259.710233:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763260.094238:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763260.094247:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763260.094251:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763260.094260:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3186 00000100:00080000:2.0:1789763260.094270:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763260.094294:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763260.094335:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504322944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763260.094372:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763260.094376:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763260.094378:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763260.094382:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3186 00000100:00080000:2.0:1789763260.094386:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763260.094389:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763260.094400:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763260.094413:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504323072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763260.094424:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763260.094428:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763260.094430:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763260.094435:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3186 00000100:00080000:2.0:1789763260.094438:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763260.094441:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763260.094445:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763260.094456:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504323200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763260.094473:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763260.094476:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504322944:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763260.094478:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763260.094483:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763260.094502:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3186 00000100:00080000:2.0:1789763260.094512:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763260.094516:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763260.094520:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763260.094536:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504323328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763260.094563:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763260.094567:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763260.094570:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763260.094574:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3186 00000100:00080000:2.0:1789763260.094580:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763260.094585:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763260.094593:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763260.094609:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504323456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763260.094620:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763260.094624:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763260.094626:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763260.094630:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3186 00000100:00080000:2.0:1789763260.094646:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763260.094650:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763260.094654:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763260.094669:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504323584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763260.094675:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763260.094776:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763260/real 1789763260] req@ffff9018d0f26d80 x1876699504322944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763260.094807:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504322944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763260.094825:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763260.094831:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763260.094837:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504322944:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763260.094848:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504323072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763260.094894:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763260.094911:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763260/real 1789763260] req@ffff9018d0f25f80 x1876699504323072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763260.094938:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504323072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763260.094948:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763260.094953:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763260.094957:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504323072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763260.094996:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504323200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763260.095018:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763260.095030:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763260/real 1789763260] req@ffff9018d0f27100 x1876699504323200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763260.095103:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504323200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763260.095122:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763260.095126:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763260.095130:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504323200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763260.095139:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504323328:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763260.095166:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763260.095214:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763260/real 1789763260] req@ffff9018d0f26300 x1876699504323328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763260.095231:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504323328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763260.095242:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763260.095246:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763260.095250:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504323328:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763260.095258:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504323456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763260.095285:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763260.095297:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763260/real 1789763260] req@ffff9018d0f25180 x1876699504323456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763260.095312:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504323456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763260.095321:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763260.095325:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763260.095328:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504323456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763260.095335:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504323584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763260.095355:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763260.095367:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763260/real 1789763260] req@ffff9018d0f24380 x1876699504323584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763260.095382:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504323584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763260.095392:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763260.095395:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763260.095399:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504323584:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763260.734305:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763261.758248:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763262.782249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763263.806291:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763264.830335:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763265.214173:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763265.214193:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763265.214202:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763265.214223:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3191 00000100:00080000:2.0:1789763265.214244:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763265.214258:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763265.214325:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504323712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763265.214375:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763265.214381:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763265.214384:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763265.214390:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3191 00000100:00080000:2.0:1789763265.214394:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763265.214400:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763265.214417:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763265.214436:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504323840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763265.214452:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763265.214456:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763265.214459:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763265.214470:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3191 00000100:00080000:2.0:1789763265.214475:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763265.214477:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504323712:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763265.214483:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763265.214499:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763265.214517:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504323968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763265.214531:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763265.214537:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763265.214540:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763265.214548:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3191 00000100:00080000:2.0:1789763265.214552:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763265.214558:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763265.214566:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763265.214579:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504324096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763265.214592:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763265.214597:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763265.214601:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789763265.214615:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763265.214632:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3191 00000100:00080000:2.0:1789763265.214656:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763265.214666:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763265.214680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763265/real 1789763265] req@ffff9018d0f24a80 x1876699504323712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763265.214687:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763265.214733:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504324224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763265.214744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504323712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763265.214791:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763265.214799:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763265.214800:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763265.214804:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763265.214814:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763265.214822:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504323712:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763265.214832:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3191 00000100:00080000:2.0:1789763265.214841:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763265.214850:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763265.214860:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763265.214870:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504323840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763265.214891:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504324352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763265.214940:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763265.214978:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763265/real 1789763265] req@ffff9018d0f26a00 x1876699504323840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763265.214996:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504323840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763265.215009:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763265.215014:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763265.215019:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504323840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763265.215028:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504323968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763265.215120:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763265.215169:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763265/real 1789763265] req@ffff9018d0f25c00 x1876699504323968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763265.215187:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504323968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763265.215198:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763265.215202:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763265.215206:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504323968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763265.215214:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504324096:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763265.215243:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763265.215257:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763265/real 1789763265] req@ffff9018d0f27800 x1876699504324096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763265.215281:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504324096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763265.215298:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763265.215305:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763265.215313:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504324096:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763265.215325:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504324224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763265.215366:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763265.215384:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763265/real 1789763265] req@ffff9018d0f24e00 x1876699504324224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763265.215400:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504324224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763265.215411:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763265.215415:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763265.215419:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504324224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763265.215449:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504324352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763265.215485:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763265.215509:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763265/real 1789763265] req@ffff9018d0f24000 x1876699504324352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763265.215535:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504324352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763265.215550:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763265.215554:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763265.215558:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504324352:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763265.854293:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763266.878229:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763267.902287:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763268.926299:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763269.950262:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763270.334153:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763270.334170:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763270.334177:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763270.334198:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3196 00000100:00080000:2.0:1789763270.334219:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763270.334233:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763270.334311:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504324480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763270.334365:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763270.334374:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763270.334378:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763270.334388:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3196 00000100:00080000:2.0:1789763270.334398:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763270.334405:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763270.334422:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763270.334447:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504324608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763270.334467:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763270.334472:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763270.334474:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763270.334486:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3196 00000100:00100000:3.0:1789763270.334486:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504324480:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763270.334498:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763270.334505:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763270.334512:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763270.334529:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504324736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763270.334542:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763270.334547:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763270.334550:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763270.334555:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3196 00000100:00080000:2.0:1789763270.334604:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763270.334609:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763270.334617:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789763270.334625:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763270.334642:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504324864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763270.334660:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763270.334678:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763270.334681:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763270.334688:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3196 00000100:00080000:2.0:1789763270.334695:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763270.334700:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763270/real 1789763270] req@ffff9018d0f25500 x1876699504324480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763270.334707:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763270.334720:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763270.334751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504324480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763270.334754:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504324992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763270.334794:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763270.334804:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763270.334807:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763270.334814:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504324480:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763270.334822:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763270.334831:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763270.334843:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3196 00000100:00080000:2.0:1789763270.334854:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763270.334859:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504324608:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763270.334869:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763270.334880:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763270.334900:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763270.334909:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504325120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763270.334925:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763270/real 1789763270] req@ffff9018d0f27480 x1876699504324608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763270.334947:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504324608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763270.334959:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763270.334964:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763270.334969:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504324608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763270.334977:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504324736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763270.335000:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763270.335011:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763270/real 1789763270] req@ffff9018d0f26680 x1876699504324736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763270.335073:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504324736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763270.335084:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763270.335088:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763270.335125:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504324736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763270.335134:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504324864:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763270.335162:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763270.335177:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763270/real 1789763270] req@ffff9018d0f24700 x1876699504324864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763270.335192:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504324864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763270.335203:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763270.335206:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763270.335210:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504324864:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763270.335217:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504324992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763270.335236:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763270.335247:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763270/real 1789763270] req@ffff9018d0f24000 x1876699504324992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763270.335261:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504324992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763270.335279:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763270.335283:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763270.335286:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504324992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763270.335310:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504325120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763270.335330:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763270.335342:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763270/real 1789763270] req@ffff9018d0f24e00 x1876699504325120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763270.335356:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504325120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763270.335366:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763270.335369:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763270.335373:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504325120:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763270.974298:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763271.998323:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763273.022312:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763274.046351:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763275.070327:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763275.454231:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763275.454244:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763275.454250:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763275.454263:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3202 00000100:00080000:1.0:1789763275.454278:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763275.454322:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763275.454373:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504325248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763275.454410:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763275.454415:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763275.454417:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763275.454423:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3202 00000100:00080000:1.0:1789763275.454428:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763275.454432:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763275.454446:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763275.454462:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504325376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763275.454476:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763275.454481:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763275.454483:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763275.454488:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3202 00000100:00080000:1.0:1789763275.454491:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763275.454497:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763275.454502:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763275.454524:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504325248:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763275.454532:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504325504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763275.454548:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763275.454553:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763275.454555:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763275.454561:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3202 00000100:00080000:1.0:1789763275.454565:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763275.454569:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763275.454574:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763275.454586:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504325632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763275.454598:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763275.454602:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763275.454604:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763275.454608:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3202 00000100:00080000:1.0:1789763275.454611:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763275.454615:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763275.454620:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763275.454630:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504325760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763275.454646:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763275.454652:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763275.454655:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763275.454661:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3202 00000100:00080000:1.0:1789763275.454667:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763275.454673:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763275.454683:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763275.454703:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504325888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763275.456521:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763275.456604:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763275/real 1789763275] req@ffff9018c66a5c00 x1876699504325248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763275.456647:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504325248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763275.456666:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763275.456673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763275.456679:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504325248:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763275.456692:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504325376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763275.456731:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763275.456750:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763275/real 1789763275] req@ffff9018c66a6d80 x1876699504325376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763275.456766:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504325376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763275.456776:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763275.456780:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763275.456784:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504325376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763275.456825:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504325504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763275.456847:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763275.456860:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763275/real 1789763275] req@ffff9018c66a5f80 x1876699504325504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763275.456873:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504325504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763275.456882:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763275.456886:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763275.456890:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504325504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763275.456897:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504325632:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763275.456917:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763275.456930:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763275/real 1789763275] req@ffff9018c66a4000 x1876699504325632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763275.456943:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504325632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763275.456952:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763275.456956:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763275.456960:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504325632:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763275.456966:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504325760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763275.456985:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763275.456997:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763275/real 1789763275] req@ffff9018c66a7b80 x1876699504325760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763275.457010:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504325760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763275.457019:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763275.457023:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763275.457026:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504325760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763275.457089:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504325888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763275.457134:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763275.457159:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763275/real 1789763275] req@ffff9018c66a7800 x1876699504325888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763275.457183:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504325888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763275.457194:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763275.457197:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763275.457201:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504325888:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763276.094267:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763277.118301:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763278.142211:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763279.166293:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763280.190262:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763280.574190:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763280.574201:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763280.574206:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763280.574224:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3207 00000100:00080000:1.0:1789763280.574238:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763280.574250:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763280.574320:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504326016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763280.574347:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763280.574351:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763280.574353:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763280.574357:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3207 00000100:00080000:1.0:1789763280.574360:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763280.574364:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763280.574374:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763280.574387:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504326144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763280.574428:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763280.574432:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504326016:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763280.574434:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763280.574445:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763280.574458:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3207 00000100:00080000:1.0:1789763280.574462:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763280.574466:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763280.574472:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763280.574487:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504326272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763280.574496:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763280.574500:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763280.574501:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763280.574505:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3207 00000100:00080000:1.0:1789763280.574508:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763280.574512:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763280.574520:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763280.574532:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504326400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763280.574545:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763280.574550:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763280.574554:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763280.574565:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3207 00000800:00000100:3.0:1789763280.574572:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763280.574576:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763280.574583:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763280.574591:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763280.574605:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504326528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763280.574647:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763280.574652:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763280.574655:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763280.574662:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3207 00000100:00080000:1.0:1789763280.574666:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763280.574671:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763280.574677:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763280.574680:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763280/real 1789763280] req@ffff9018c66a5180 x1876699504326016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763280.574699:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504326656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763280.574733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504326016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763280.574759:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763280.574765:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763280.574771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504326016:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763280.574818:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504326144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763280.574851:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763280.574868:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763280/real 1789763280] req@ffff9018c66a6300 x1876699504326144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763280.574884:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504326144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763280.574895:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763280.574900:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763280.574904:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504326144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763280.574913:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504326272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763280.574933:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763280.574945:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763280/real 1789763280] req@ffff9018c66a4700 x1876699504326272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763280.574959:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504326272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763280.574968:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763280.574972:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763280.574976:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504326272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763280.574994:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504326400:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763280.575015:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763280.575116:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763280/real 1789763280] req@ffff9018c66a6680 x1876699504326400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763280.575143:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504326400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763280.575159:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763280.575166:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763280.575170:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504326400:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763280.575179:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504326528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763280.575212:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763280.575227:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763280/real 1789763280] req@ffff9018c66a5500 x1876699504326528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763280.575242:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504326528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763280.575252:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763280.575256:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763280.575260:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504326528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763280.575267:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504326656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763280.575286:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763280.575298:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763280/real 1789763280] req@ffff9018c66a4e00 x1876699504326656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763280.575312:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504326656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763280.575321:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763280.575325:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763280.575328:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504326656:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763281.214187:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763282.238152:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763283.262204:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763284.286238:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763285.310196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763285.694155:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763285.694165:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763285.694171:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763285.694183:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3212 00000100:00080000:2.0:1789763285.694195:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763285.694202:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763285.694243:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504326784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763285.694275:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763285.694279:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763285.694280:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763285.694286:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3212 00000100:00080000:2.0:1789763285.694289:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763285.694294:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763285.694305:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763285.694327:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504326912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763285.694332:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504326784:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763285.694337:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763285.694352:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763285.694354:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763285.694357:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3212 00000100:00080000:2.0:1789763285.694359:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763285.694363:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763285.694366:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763285.694376:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504327040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763285.694383:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763285.694385:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763285.694386:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763285.694389:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3212 00000100:00080000:2.0:1789763285.694470:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763285.694473:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763285.694478:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789763285.694482:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763285.694493:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504327168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763285.694502:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763285.694504:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763285.694505:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763285.694508:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3212 00000100:00080000:2.0:1789763285.694509:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763285.694514:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763285.694517:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763285.694523:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504327296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763285.694528:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763285.694531:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763285.694532:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763285.694535:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3212 00000100:00080000:2.0:1789763285.694537:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763285.694539:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763285.694542:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763285.694552:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504327424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763285.694664:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763285/real 1789763285] req@ffff9018d0f27800 x1876699504326784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763285.694690:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504326784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763285.694703:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763285.694708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763285.694713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504326784:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763285.694741:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504326912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763285.694759:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763285.694768:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763285/real 1789763285] req@ffff9018d0f25c00 x1876699504326912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763285.694777:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504326912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763285.694783:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763285.694786:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763285.694788:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504326912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763285.694792:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504327040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763285.694803:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763285.694809:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763285/real 1789763285] req@ffff9018d0f26a00 x1876699504327040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763285.694817:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504327040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763285.694827:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763285.694829:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763285.694831:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504327040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763285.694835:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504327168:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763285.694846:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763285.694852:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763285/real 1789763285] req@ffff9018d0f24a80 x1876699504327168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763285.694858:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504327168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763285.694864:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763285.694866:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763285.694868:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504327168:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763285.694872:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504327296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763285.694881:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763285.694887:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763285/real 1789763285] req@ffff9018d0f24380 x1876699504327296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763285.694901:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504327296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763285.694906:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763285.694908:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763285.694911:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504327296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763285.694915:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504327424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763285.694926:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763285.694933:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763285/real 1789763285] req@ffff9018d0f25180 x1876699504327424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763285.694941:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504327424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763285.694946:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763285.694948:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763285.694951:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504327424:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763286.334164:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763287.358205:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763288.382206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763289.406166:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763290.430189:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763290.814131:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763290.814139:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763290.814143:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763290.814151:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3217 00000100:00080000:3.0:1789763290.814160:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763290.814167:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763290.814200:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504327552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763290.814224:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763290.814227:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763290.814229:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763290.814232:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3217 00000100:00080000:3.0:1789763290.814236:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763290.814240:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763290.814249:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763290.814262:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504327680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763290.814263:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504327552:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763290.814279:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763290.814281:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763290.814283:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763290.814285:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3217 00000100:00080000:3.0:1789763290.814287:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763290.814289:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763290.814292:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763290.814300:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504327808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763290.814306:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763290.814308:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763290.814309:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763290.814312:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3217 00000100:00080000:3.0:1789763290.814326:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763290.814329:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763290.814334:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763290.814343:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504327936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763290.814356:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763290.814359:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763290.814361:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763290.814363:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3217 00000100:00080000:3.0:1789763290.814365:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763290.814368:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763290.814373:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504327680:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789763290.814373:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763290.814386:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504328064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763290.814396:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763290.814399:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763290.814401:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763290.814404:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3217 00000100:00100000:2.0:1789763290.814407:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504327808:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789763290.814407:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763290.814412:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763290.814416:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763290.814427:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504328192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763290.814435:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504327936:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763290.814456:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504328064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763290.814471:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504328192:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763291.454178:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763292.478225:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763293.502207:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763293.886342:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:1.0:1789763293.886399:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789763293.886421:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763293.886467:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763293.886479:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763293.886489:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763293.886498:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763293.886508:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:2.0:1789763293.886551:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763290/real 1789763293] req@ffff9018d070f480 x1876699504328064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763293.886585:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504328064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763293.886602:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763293.886609:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763293.886614:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504328064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763293.886625:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763290/real 1789763293] req@ffff9018d070fb80 x1876699504328192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763293.886640:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504328192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763293.886650:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763293.886654:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763293.886659:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504328192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763293.886667:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763290/real 1789763293] req@ffff9018d070df80 x1876699504327680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763293.886680:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504327680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763293.886689:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763293.886693:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763293.886696:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504327680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763293.886704:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763290/real 1789763293] req@ffff9018d070ce00 x1876699504327808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763293.886717:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504327808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763293.886726:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763293.886731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763293.886734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504327808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763293.886742:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763290/real 1789763293] req@ffff9018d070ed80 x1876699504327936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763293.886754:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504327936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763293.886763:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763293.886767:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763293.886770:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504327936:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763293.886777:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763290/real 1789763293] req@ffff9018d070d880 x1876699504327552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763293.886790:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504327552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763293.886798:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763293.886802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763293.886806:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504327552:192.168.203.158@tcp:250:kworker.0 00010000:00010000:3.0:1789763294.526165:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763295.550188:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763295.934110:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763295.934121:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763295.934125:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763295.934136:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3222 00000100:00080000:3.0:1789763295.934147:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763295.934155:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763295.934201:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504328320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763295.934233:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763295.934237:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763295.934239:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763295.934243:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3222 00000100:00080000:3.0:1789763295.934247:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763295.934251:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763295.934263:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763295.934279:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504328448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763295.934291:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763295.934293:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504328320:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763295.934296:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763295.934300:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763295.934307:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3222 00000100:00080000:3.0:1789763295.934311:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763295.934316:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763295.934321:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763295.934335:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504328576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763295.934352:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763295.934356:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763295.934358:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763295.934363:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3222 00000100:00080000:3.0:1789763295.934375:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763295.934379:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763295.934385:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789763295.934393:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763295.934399:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504328704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763295.934412:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763295.934416:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763295.934418:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763295.934422:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3222 00000100:00080000:3.0:1789763295.934426:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763295.934430:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763295.934436:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763295.934446:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763295/real 1789763295] req@ffff9018d070ca80 x1876699504328320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763295.934451:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504328832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763295.934479:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763295.934483:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763295.934486:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763295.934490:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504328320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763295.934494:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3222 00000100:00080000:3.0:1789763295.934502:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763295.934509:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789763295.934511:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789763295.934516:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763295.934520:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763295.934527:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504328320:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763295.934534:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504328960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763295.934562:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504328448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763295.934598:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763295.934613:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763295/real 1789763295] req@ffff9018d070d180 x1876699504328448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763295.934634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504328448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763295.934648:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763295.934652:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763295.934655:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504328448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763295.934663:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504328576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763295.934687:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763295.934700:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763295/real 1789763295] req@ffff9018d070e680 x1876699504328576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763295.934715:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504328576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763295.934724:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763295.934728:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763295.934732:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504328576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763295.934739:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504328704:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763295.934763:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763295.934775:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763295/real 1789763295] req@ffff9018d070f100 x1876699504328704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763295.934803:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504328704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763295.934813:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763295.934815:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763295.934817:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504328704:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763295.934822:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504328832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763295.934843:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763295.934855:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763295/real 1789763295] req@ffff9018d070f800 x1876699504328832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763295.934868:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504328832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763295.934877:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763295.934881:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763295.934884:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504328832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763295.934891:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504328960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763295.934926:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763295.934939:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763295/real 1789763295] req@ffff9018d070d500 x1876699504328960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763295.934951:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504328960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763295.934965:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763295.934969:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763295.934972:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504328960:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763296.574166:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763297.598145:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763298.622163:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763299.646280:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763300.670201:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763301.054147:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763301.054157:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763301.054162:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763301.054173:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3227 00000100:00080000:0.0:1789763301.054186:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763301.054193:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763301.054240:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504329088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763301.054267:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763301.054270:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763301.054271:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763301.054274:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3227 00000100:00080000:0.0:1789763301.054277:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763301.054280:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763301.054293:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763301.054310:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504329216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763301.054324:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763301.054328:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763301.054330:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763301.054335:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504329088:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763301.054337:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3227 00000100:00080000:0.0:1789763301.054344:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763301.054348:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763301.054353:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763301.054364:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504329344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763301.054371:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763301.054374:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763301.054375:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763301.054379:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3227 00000100:00080000:0.0:1789763301.054384:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763301.054387:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763301.054391:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763301.054399:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504329472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763301.054405:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763301.054409:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1789763301.054411:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763301.054412:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763301.054419:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3227 00000100:00080000:0.0:1789763301.054422:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763301.054427:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763301.054433:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763301.054446:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504329600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763301.054457:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763301.054461:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763301/real 1789763301] req@ffff9018c8b98e00 x1876699504329088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763301.054464:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763301.054468:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763301.054477:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3227 00000100:00080000:0.0:1789763301.054481:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763301.054484:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504329088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763301.054488:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763301.054495:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763301.054500:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763301.054506:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763301.054509:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504329728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763301.054513:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504329088:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763301.054541:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504329216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763301.054559:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763301.054567:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763301/real 1789763301] req@ffff9018c8b9aa00 x1876699504329216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763301.054576:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504329216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763301.054581:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763301.054584:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763301.054587:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504329216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763301.054591:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504329344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763301.054602:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763301.054607:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763301/real 1789763301] req@ffff9018c8b98000 x1876699504329344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763301.054614:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504329344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763301.054620:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763301.054621:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763301.054623:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504329344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763301.054626:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504329472:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763301.054637:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763301.054642:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763301/real 1789763301] req@ffff9018c8b9b800 x1876699504329472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763301.054649:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504329472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763301.054654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763301.054656:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763301.054658:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504329472:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763301.054661:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504329600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763301.054671:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763301.054677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763301/real 1789763301] req@ffff9018c8b99880 x1876699504329600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763301.054684:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504329600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763301.054690:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763301.054691:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763301.054693:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504329600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763301.054696:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504329728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763301.054706:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763301.054711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763301/real 1789763301] req@ffff9018c8b98a80 x1876699504329728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763301.054717:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504329728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763301.054722:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763301.054724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763301.054726:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504329728:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763301.694159:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763302.718186:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763303.742172:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763304.766213:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763305.790190:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763306.174151:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763306.174159:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763306.174163:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763306.174171:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3232 00000100:00080000:3.0:1789763306.174180:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763306.174186:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763306.174219:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504329856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763306.174245:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763306.174247:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763306.174249:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763306.174252:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3232 00000100:00080000:3.0:1789763306.174254:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763306.174257:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763306.174266:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763306.174281:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504329984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763306.174290:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504329856:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763306.174299:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763306.174305:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763306.174307:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763306.174313:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3232 00000100:00080000:3.0:1789763306.174317:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763306.174320:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763306.174324:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763306.174335:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504330112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763306.174343:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763306.174346:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763306.174350:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789763306.174351:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763306.174357:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3232 00000100:00080000:3.0:1789763306.174361:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763306.174364:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763306.174368:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763306.174377:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504330240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763306.174385:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763306.174388:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763306.174389:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763306.174391:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3232 00000100:00080000:3.0:1789763306.174394:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763306.174398:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763306.174402:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763306/real 1789763306] req@ffff9018d070dc00 x1876699504329856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789763306.174404:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763306.174416:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504330368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763306.174439:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763306.174440:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504329856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763306.174444:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763306.174447:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763306.174454:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3232 00000100:00080000:3.0:1789763306.174459:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763306.174462:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789763306.174464:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789763306.174465:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763306.174473:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763306.174480:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504329856:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763306.174482:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504330496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763306.174516:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504329984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763306.174542:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763306.174552:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763306/real 1789763306] req@ffff9018d070c000 x1876699504329984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763306.174561:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504329984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763306.174567:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763306.174569:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763306.174572:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504329984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763306.174576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504330112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763306.174587:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763306.174593:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763306/real 1789763306] req@ffff9018d070e300 x1876699504330112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763306.174601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504330112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763306.174610:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763306.174612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763306.174614:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504330112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763306.174618:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504330240:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763306.174628:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763306.174634:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763306/real 1789763306] req@ffff9018d070d500 x1876699504330240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763306.174640:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504330240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763306.174646:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763306.174648:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763306.174649:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504330240:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763306.174653:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504330368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763306.174673:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763306.174679:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763306/real 1789763306] req@ffff9018d070f800 x1876699504330368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763306.174687:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504330368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763306.174692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763306.174694:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763306.174696:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504330368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763306.174700:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504330496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763306.174710:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763306.174716:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763306/real 1789763306] req@ffff9018d070f100 x1876699504330496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763306.174722:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504330496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763306.174727:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763306.174729:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763306.174731:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504330496:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763306.814164:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763307.838157:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763308.862141:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763309.886164:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763310.910249:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763311.294174:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763311.294189:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763311.294195:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763311.294210:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3237 00000100:00080000:3.0:1789763311.294227:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763311.294237:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763311.294297:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504330624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763311.294353:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763311.294360:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763311.294363:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763311.294372:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3237 00000100:00080000:3.0:1789763311.294379:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763311.294386:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763311.294405:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763311.294431:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504330752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763311.294457:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504330624:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763311.294459:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763311.294477:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763311.294480:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763311.294487:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3237 00000100:00080000:3.0:1789763311.294494:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763311.294500:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763311.294508:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763311.294546:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504330880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763311.294558:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763311.294562:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763311.294564:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763311.294569:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3237 00000100:00080000:3.0:1789763311.294574:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763311.294579:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789763311.294586:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789763311.294587:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763311.294606:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504331008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763311.294618:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763311.294622:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763311.294624:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763311.294628:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3237 00000100:00080000:3.0:1789763311.294633:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763311.294639:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763311.294644:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763311.294651:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763311/real 1789763311] req@ffff9018d070e680 x1876699504330624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763311.294678:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504331136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763311.294681:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504330624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763311.294715:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763311.294721:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763311.294726:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763311.294730:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789763311.294732:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763311.294739:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504330624:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763311.294742:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3237 00000100:00080000:3.0:1789763311.294752:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763311.294758:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763311.294763:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763311.294776:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504331264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763311.294783:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504330752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763311.294895:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763311.294926:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763311/real 1789763311] req@ffff9018d070d180 x1876699504330752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763311.294953:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504330752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763311.294970:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763311.294978:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763311.294985:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504330752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763311.294999:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504330880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763311.295039:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763311.295100:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763311/real 1789763311] req@ffff9018d070ca80 x1876699504330880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763311.295117:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504330880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763311.295128:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763311.295132:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763311.295136:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504330880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763311.295144:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504331008:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763311.295170:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763311.295183:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763311/real 1789763311] req@ffff9018d070d880 x1876699504331008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763311.295206:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504331008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763311.295216:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763311.295220:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763311.295224:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504331008:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763311.295234:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504331136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763311.295269:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763311.295292:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763311/real 1789763311] req@ffff9018d070ed80 x1876699504331136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763311.295317:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504331136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763311.295334:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763311.295341:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763311.295346:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504331136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763311.295378:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504331264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763311.295407:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763311.295421:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763311/real 1789763311] req@ffff9018d070ce00 x1876699504331264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763311.295435:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504331264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763311.295453:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763311.295457:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763311.295461:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504331264:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763311.934232:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763312.958289:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763313.982334:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763315.006291:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763316.030258:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763316.414176:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763316.414189:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763316.414195:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763316.414206:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3242 00000100:00080000:2.0:1789763316.414222:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763316.414234:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763316.414291:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504331392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763316.414340:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763316.414348:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763316.414352:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763316.414360:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3242 00000100:00080000:2.0:1789763316.414369:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763316.414377:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763316.414394:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763316.414422:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504331520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763316.414443:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504331392:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763316.414469:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763316.414475:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763316.414478:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763316.414484:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3242 00000100:00080000:2.0:1789763316.414489:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763316.414495:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763316.414503:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763316.414538:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504331648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763316.414559:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763316.414581:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763316.414589:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763316.414594:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763316.414603:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3242 00000100:00080000:2.0:1789763316.414611:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763316.414620:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763316.414630:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763316/real 1789763316] req@ffff9018d0f26300 x1876699504331392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763316.414631:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763316.414666:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504331776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763316.414671:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504331392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763316.414738:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763316.414741:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763316.414745:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763316.414773:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763316.414780:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763316.414780:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504331392:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763316.414791:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3242 00000100:00080000:2.0:1789763316.414813:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763316.414827:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763316.414832:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763316.414842:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504331904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763316.414849:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504331520:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763316.414852:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763316.414858:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763316.414860:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763316.414863:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3242 00000100:00080000:2.0:1789763316.414866:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763316.414868:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763316.414871:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763316.414882:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504332032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763316.414882:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763316.414898:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763316/real 1789763316] req@ffff9018d0f27100 x1876699504331520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763316.414910:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504331520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763316.414917:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763316.414920:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763316.414923:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504331520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763316.414927:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504331648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763316.414939:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763316.414945:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763316/real 1789763316] req@ffff9018d0f25f80 x1876699504331648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763316.414953:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504331648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763316.414959:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763316.414960:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763316.414962:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504331648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763316.414966:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504331776:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763316.414976:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763316.414982:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763316/real 1789763316] req@ffff9018d0f26d80 x1876699504331776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763316.414989:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504331776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763316.414995:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763316.414997:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763316.414999:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504331776:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763316.415009:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504331904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763316.415018:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763316.415024:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763316/real 1789763316] req@ffff9018d0f26300 x1876699504331904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763316.415036:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504331904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763316.415042:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763316.415043:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763316.415045:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504331904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763316.415066:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504332032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763316.415078:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763316.415100:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763316/real 1789763316] req@ffff9018d0f25180 x1876699504332032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763316.415109:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504332032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763316.415115:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763316.415117:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763316.415119:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504332032:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763317.054288:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763318.078261:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763319.102290:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763320.126269:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763321.150246:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763321.534149:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763321.534158:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763321.534161:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763321.534169:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3248 00000100:00080000:2.0:1789763321.534179:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763321.534184:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763321.534216:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504332160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763321.534239:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763321.534242:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763321.534244:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763321.534247:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3248 00000100:00080000:2.0:1789763321.534249:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763321.534252:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763321.534260:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763321.534268:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504332288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763321.534275:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763321.534277:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763321.534278:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763321.534280:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3248 00000100:00080000:2.0:1789763321.534283:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763321.534286:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763321.534288:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763321.534296:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504332416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763321.534302:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763321.534305:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763321.534306:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763321.534309:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3248 00000100:00080000:2.0:1789763321.534310:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763321.534320:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763321.534325:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763321.534337:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504332544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763321.534349:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504332160:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763321.534350:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763321.534363:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763321.534368:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763321.534372:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3248 00000100:00080000:2.0:1789763321.534374:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763321.534377:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763321.534380:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763321.534390:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504332672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763321.534396:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763321.534398:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763321.534400:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763321.534403:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3248 00000100:00080000:2.0:1789763321.534405:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763321.534408:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763321.534411:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763321.534418:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504332800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763321.534474:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763321.534542:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763321/real 1789763321] req@ffff9018d0f24380 x1876699504332160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763321.534578:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504332160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763321.534597:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763321.534605:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763321.534614:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504332160:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763321.534627:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504332288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763321.534665:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763321.534682:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763321/real 1789763321] req@ffff9018d0f24a80 x1876699504332288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763321.534700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504332288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763321.534712:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763321.534746:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763321.534756:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504332288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763321.534772:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504332416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763321.534814:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763321.534838:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763321/real 1789763321] req@ffff9018d0f26a00 x1876699504332416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763321.534872:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504332416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763321.534895:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763321.534902:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763321.534909:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504332416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763321.534959:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504332544:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763321.535373:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763321.535467:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763321/real 1789763321] req@ffff9018d0f25c00 x1876699504332544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763321.535489:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504332544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763321.535501:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763321.535506:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763321.535510:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504332544:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763321.535519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504332672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763321.535546:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763321.535561:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763321/real 1789763321] req@ffff9018d0f27800 x1876699504332672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763321.535575:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504332672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763321.535584:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763321.535588:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763321.535591:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504332672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763321.535598:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504332800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763321.535618:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763321.535629:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763321/real 1789763321] req@ffff9018d0f24e00 x1876699504332800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763321.535644:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504332800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763321.535653:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763321.535656:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763321.535660:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504332800:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763322.174141:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763323.198183:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763324.222244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763325.246308:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763326.270269:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763326.654235:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763326.654250:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763326.654257:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763326.654274:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3253 00000100:00080000:0.0:1789763326.654292:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763326.654304:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763326.654368:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504332928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763326.654415:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763326.654423:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763326.654427:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763326.654435:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3253 00000100:00080000:0.0:1789763326.654442:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763326.654450:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763326.654469:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763326.654494:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504333056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763326.654528:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763326.654532:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504332928:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763326.654534:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763326.654538:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763326.654545:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3253 00000100:00080000:0.0:1789763326.654551:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763326.654558:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763326.654566:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763326.654584:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504333184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763326.654599:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763326.654605:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763326.654608:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763326.654614:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3253 00000100:00080000:0.0:1789763326.654619:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763326.654626:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763326.654634:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763326.654651:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504333312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763326.654651:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763326.654669:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763326.654675:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763326.654707:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763326.654716:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3253 00000100:00080000:0.0:1789763326.654723:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763326.654730:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763326.654739:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763326.654759:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504333440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763326.654767:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763326/real 1789763326] req@ffff9018c8b9b480 x1876699504332928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763326.654792:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763326.654798:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763326.654801:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763326.654806:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504332928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763326.654809:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3253 00000100:00080000:0.0:1789763326.654816:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763326.654822:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789763326.654827:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789763326.654831:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789763326.654835:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763326.654842:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504332928:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763326.654851:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504333568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763326.654886:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504333056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763326.654934:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763326.654961:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763326/real 1789763326] req@ffff9018c8b9b100 x1876699504333056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763326.654978:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504333056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763326.655003:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763326.655008:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763326.655012:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504333056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763326.655021:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504333184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763326.655091:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763326.655120:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763326/real 1789763326] req@ffff9018c8b9bb80 x1876699504333184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763326.655136:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504333184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763326.655146:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763326.655150:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763326.655154:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504333184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763326.655162:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504333312:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763326.655186:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763326.655198:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763326/real 1789763326] req@ffff9018c8b98700 x1876699504333312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763326.655212:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504333312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763326.655222:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763326.655225:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763326.655230:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504333312:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763326.655237:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504333440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763326.655256:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763326.655268:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763326/real 1789763326] req@ffff9018c8b99f80 x1876699504333440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763326.655281:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504333440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763326.655291:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763326.655294:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763326.655298:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504333440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763326.655304:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504333568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763326.655323:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763326.655333:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763326/real 1789763326] req@ffff9018c8b98a80 x1876699504333568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763326.655347:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504333568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763326.655357:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763326.655360:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763326.655364:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504333568:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763327.294298:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763328.318249:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763329.342185:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763330.366211:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763331.390297:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763331.774236:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763331.774248:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763331.774253:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763331.774266:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3258 00000100:00080000:1.0:1789763331.774280:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763331.774289:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763331.774358:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504333696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763331.774397:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763331.774403:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763331.774405:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763331.774410:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3258 00000100:00080000:1.0:1789763331.774414:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763331.774419:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763331.774433:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763331.774450:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504333824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763331.774465:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763331.774471:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763331.774477:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763331.774480:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504333696:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763331.774492:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3258 00000100:00080000:1.0:1789763331.774501:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763331.774509:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763331.774520:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763331.774551:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504333952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763331.774574:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789763331.774579:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763331.774589:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763331.774594:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763331.774603:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3258 00000100:00080000:1.0:1789763331.774612:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763331.774618:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763331.774627:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763331.774669:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763331/real 1789763331] req@ffff9018c66a5880 x1876699504333696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763331.774682:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504334080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763331.774709:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504333696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763331.774713:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763331.774732:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763331.774741:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763331.774743:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763331.774759:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789763331.774764:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3258 00000100:00080000:1.0:1789763331.774775:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763331.774775:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504333696:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763331.774791:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763331.774802:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763331.774820:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504333824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763331.774827:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504334208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763331.774846:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763331.774854:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763331.774860:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789763331.774864:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763331.774875:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3258 00000100:00080000:1.0:1789763331.774887:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763331.774892:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763331/real 1789763331] req@ffff9018c66a7480 x1876699504333824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763331.774903:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763331.774924:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763331.774926:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504333824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763331.774945:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763331.774951:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763331.774960:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504334336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763331.774961:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504333824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763331.774982:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504333952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763331.775025:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763331.775114:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763331/real 1789763331] req@ffff9018c66a4a80 x1876699504333952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763331.775157:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504333952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763331.775173:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763331.775179:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763331.775185:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504333952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763331.775199:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504334080:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763331.775262:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763331.775289:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763331/real 1789763331] req@ffff9018c66a7100 x1876699504334080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763331.775310:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504334080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763331.775321:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763331.775325:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763331.775329:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504334080:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763331.775353:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504334208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763331.775382:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763331.775395:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763331/real 1789763331] req@ffff9018c66a4380 x1876699504334208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763331.775410:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504334208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763331.775430:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763331.775439:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763331.775445:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504334208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763331.775453:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504334336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763331.775479:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763331.775494:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763331/real 1789763331] req@ffff9018c66a6a00 x1876699504334336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763331.775508:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504334336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763331.775518:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763331.775522:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763331.775526:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504334336:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763332.414287:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763333.438238:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763334.462273:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763335.486236:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763336.510278:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763336.894230:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763336.894253:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763336.894259:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763336.894270:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3263 00000100:00080000:0.0:1789763336.894284:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763336.894293:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763336.894363:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504334464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763336.894400:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763336.894405:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763336.894407:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763336.894412:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3263 00000100:00080000:0.0:1789763336.894417:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763336.894421:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763336.894444:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763336.894462:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504334592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763336.894477:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504334464:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763336.894481:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763336.894495:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763336.894498:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763336.894503:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3263 00000100:00080000:0.0:1789763336.894507:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763336.894511:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763336.894518:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763336.894537:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504334720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763336.894549:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763336.894553:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763336.894555:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763336.894561:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3263 00000100:00080000:0.0:1789763336.894567:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763336.894573:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789763336.894582:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789763336.894587:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763336.894691:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763336/real 1789763336] req@ffff9018c8b99880 x1876699504334464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763336.894708:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504334848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763336.894726:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763336.894728:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504334464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763336.894738:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763336.894747:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763336.894763:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3263 00000100:00080000:3.0:1789763336.894765:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763336.894770:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763336.894778:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789763336.894786:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763336.894796:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504334464:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789763336.894800:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763336.894824:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504334976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763336.894839:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763336.894843:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504334592:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763336.894845:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763336.894852:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763336.894866:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3263 00000100:00080000:0.0:1789763336.894871:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763336.894875:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763336.894882:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763336.894900:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504335104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763336.894943:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763336.894972:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763336/real 1789763336] req@ffff9018c8b9b800 x1876699504334592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763336.894992:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504334592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763336.895004:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763336.895009:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763336.895013:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504334592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763336.895023:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504334720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763336.895091:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763336.895107:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763336/real 1789763336] req@ffff9018c8b98000 x1876699504334720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763336.895123:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504334720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763336.895134:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763336.895139:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763336.895143:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504334720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763336.895151:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504334848:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763336.895172:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763336.895185:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763336/real 1789763336] req@ffff9018c8b9aa00 x1876699504334848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763336.895201:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504334848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763336.895216:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763336.895223:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763336.895229:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504334848:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763336.895266:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504334976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763336.895307:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763336.895329:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763336/real 1789763336] req@ffff9018c8b98e00 x1876699504334976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763336.895346:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504334976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763336.895356:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763336.895361:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763336.895365:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504334976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763336.895372:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504335104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763336.895395:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763336.895407:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763336/real 1789763336] req@ffff9018c8b9a680 x1876699504335104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763336.895422:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504335104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763336.895440:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763336.895444:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763336.895449:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504335104:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763337.534155:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763338.558152:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763339.582243:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763340.606248:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763341.630238:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763342.014266:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763342.014279:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763342.014284:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763342.014297:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3268 00000100:00080000:3.0:1789763342.014310:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763342.014319:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763342.014368:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504335232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763342.014410:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763342.014415:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763342.014418:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763342.014423:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3268 00000100:00080000:3.0:1789763342.014427:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763342.014432:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763342.014444:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763342.014459:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504335360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763342.014474:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763342.014480:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763342.014483:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763342.014489:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3268 00000100:00080000:3.0:1789763342.014493:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763342.014503:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763342.014505:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504335232:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789763342.014510:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763342.014527:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504335488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763342.014541:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763342.014545:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763342.014548:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763342.014554:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3268 00000100:00080000:3.0:1789763342.014559:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763342.014563:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763342.014593:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763342.014611:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504335616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763342.014625:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763342.014634:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763342.014637:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789763342.014645:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763342.014646:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3268 00000100:00080000:3.0:1789763342.014661:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763342.014671:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763342.014683:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763342.014711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763342/real 1789763342] req@ffff9018d070df80 x1876699504335232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763342.014716:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504335744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763342.014766:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763342.014769:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504335232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763342.014783:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763342.014793:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763342.014800:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763342.014806:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3268 00000100:00080000:2.0:1789763342.014815:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789763342.014818:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763342.014824:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504335232:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763342.014833:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763342.014844:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504335360:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789763342.014846:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763342.014883:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504335872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763342.014902:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763342.014922:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763342/real 1789763342] req@ffff9018d070fb80 x1876699504335360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763342.014940:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504335360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763342.014953:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763342.014958:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763342.014962:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504335360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763342.014996:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504335488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763342.015021:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763342.015078:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763342/real 1789763342] req@ffff9018d070f480 x1876699504335488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763342.015095:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504335488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763342.015105:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763342.015109:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763342.015113:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504335488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763342.015121:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504335616:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763342.015146:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763342.015159:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763342/real 1789763342] req@ffff9018d070ce00 x1876699504335616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763342.015173:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504335616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763342.015183:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763342.015186:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763342.015190:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504335616:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763342.015197:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504335744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763342.015215:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763342.015226:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763342/real 1789763342] req@ffff9018d070ed80 x1876699504335744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763342.015259:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504335744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763342.015269:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763342.015272:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763342.015275:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504335744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763342.015282:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504335872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763342.015300:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763342.015319:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763342/real 1789763342] req@ffff9018d070d880 x1876699504335872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763342.015340:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504335872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763342.015367:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763342.015374:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763342.015380:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504335872:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763342.654259:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763343.678240:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763344.702231:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763345.726301:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763346.750264:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763347.134205:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763347.134221:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763347.134238:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763347.134252:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3273 00000100:00080000:3.0:1789763347.134267:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763347.134278:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763347.134347:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504336000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763347.134389:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763347.134394:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763347.134398:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763347.134415:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3273 00000100:00080000:3.0:1789763347.134419:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763347.134424:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763347.134439:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763347.134454:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504336128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763347.134471:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504336000:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763347.134476:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763347.134483:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763347.134485:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763347.134495:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3273 00000100:00080000:3.0:1789763347.134501:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763347.134505:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763347.134510:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763347.134555:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504336256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763347.134580:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763347.134585:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763347.134587:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763347.134592:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3273 00000100:00080000:3.0:1789763347.134598:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763347.134599:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763347.134606:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763347.134615:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763347.134638:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504336384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763347.134652:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763347.134653:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763347/real 1789763347] req@ffff9018d070ca80 x1876699504336000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763347.134669:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763347.134680:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763347.134698:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3273 00000100:00080000:3.0:1789763347.134706:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763347.134708:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504336000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763347.134718:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763347.134735:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763347.134740:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763347.134747:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763347.134755:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504336512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763347.134761:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504336000:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763347.134794:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763347.134799:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763347.134802:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763347.134811:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504336128:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789763347.134812:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3273 00000100:00080000:3.0:1789763347.134820:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763347.134830:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763347.134837:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763347.134857:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504336640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763347.134858:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763347.134878:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763347/real 1789763347] req@ffff9018d070d180 x1876699504336128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763347.134893:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504336128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763347.134901:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763347.134907:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763347.134913:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504336128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763347.134919:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504336256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763347.134932:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763347.134938:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763347/real 1789763347] req@ffff9018d070e680 x1876699504336256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763347.134946:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504336256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763347.134951:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763347.134953:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763347.134955:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504336256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763347.134960:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504336384:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763347.134971:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763347.134977:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763347/real 1789763347] req@ffff9018d070f100 x1876699504336384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763347.134984:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504336384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763347.134990:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763347.134992:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763347.134994:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504336384:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763347.134997:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504336512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763347.135007:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763347.135013:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763347/real 1789763347] req@ffff9018d070f800 x1876699504336512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763347.135020:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504336512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763347.135025:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763347.135027:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763347.135029:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504336512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763347.135098:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504336640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763347.135119:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763347.135128:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763347/real 1789763347] req@ffff9018d070d500 x1876699504336640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763347.135136:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504336640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763347.135142:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763347.135144:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763347.135146:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504336640:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763347.774295:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763348.798275:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763349.822219:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763350.846275:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763351.870279:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763352.254195:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763352.254208:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763352.254213:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763352.254228:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3278 00000100:00080000:1.0:1789763352.254244:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763352.254256:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763352.254318:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504336768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763352.254358:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763352.254363:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763352.254366:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763352.254371:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3278 00000100:00080000:1.0:1789763352.254377:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763352.254384:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763352.254398:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763352.254424:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504336768:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763352.254436:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504336896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763352.254454:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763352.254459:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763352.254462:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763352.254468:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3278 00000100:00080000:1.0:1789763352.254473:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763352.254478:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763352.254486:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763352.254531:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504337024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763352.254563:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763352.254565:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763352.254569:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763352.254571:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763352.254575:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3278 00000100:00080000:1.0:1789763352.254580:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763352.254583:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763352.254588:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763352.254600:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504337152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763352.254603:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763352/real 1789763352] req@ffff9018c66a6a00 x1876699504336768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763352.254629:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763352.254633:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763352.254637:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763352.254637:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504336768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763352.254644:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3278 00000100:00080000:1.0:1789763352.254650:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763352.254656:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789763352.254658:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1789763352.254661:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763352.254665:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763352.254670:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504336768:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763352.254674:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504337280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763352.254683:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763352.254686:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763352.254688:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763352.254691:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3278 00000100:00080000:1.0:1789763352.254694:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763352.254697:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504336896:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789763352.254699:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763352.254707:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763352.254718:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504337408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763352.254726:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763352.254740:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763352/real 1789763352] req@ffff9018c66a4380 x1876699504336896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763352.254751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504336896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763352.254759:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763352.254762:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763352.254764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504336896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763352.254768:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504337024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763352.254780:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763352.254786:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763352/real 1789763352] req@ffff9018c66a7100 x1876699504337024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763352.254793:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504337024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763352.254799:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763352.254801:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763352.254803:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504337024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763352.254806:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504337152:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763352.254818:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763352.254823:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763352/real 1789763352] req@ffff9018c66a4a80 x1876699504337152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763352.254831:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504337152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763352.254836:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763352.254838:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763352.254841:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504337152:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763352.254844:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504337280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763352.254864:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763352.254875:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763352/real 1789763352] req@ffff9018c66a7480 x1876699504337280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763352.254884:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504337280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763352.254890:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763352.254892:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763352.254895:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504337280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763352.254910:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504337408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763352.254923:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763352.254929:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763352/real 1789763352] req@ffff9018c66a5880 x1876699504337408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763352.254939:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504337408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763352.254945:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763352.254948:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763352.254951:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504337408:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763352.894230:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763353.918428:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763354.942366:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763355.966180:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763356.990290:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763357.374268:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763357.374283:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763357.374302:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763357.374320:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3283 00000100:00080000:2.0:1789763357.374338:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763357.374350:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763357.374428:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504337536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763357.374508:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763357.374515:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763357.374518:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763357.374529:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3283 00000100:00080000:2.0:1789763357.374537:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763357.374546:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763357.374562:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763357.374576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504337536:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763357.374599:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504337664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763357.374625:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763357.374632:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763357.374635:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763357.374641:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3283 00000100:00080000:2.0:1789763357.374647:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763357.374652:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763357.374659:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763357.374681:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504337792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763357.374693:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763357.374697:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763357.374699:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763357.374704:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3283 00000100:00080000:2.0:1789763357.374709:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763357.374715:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763357.374722:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763357.374743:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504337664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763357.374744:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504337920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763357.374776:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763357.374781:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763357.374783:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763357.374790:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3283 00000100:00080000:2.0:1789763357.374797:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763357.374804:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763357.374810:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763357.374825:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504338048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763357.374833:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504337792:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763357.374848:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763357.374855:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763357.374858:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763357.374866:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3283 00000100:00080000:2.0:1789763357.374873:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763357.374887:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763357.374893:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763357.374907:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504337920:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763357.374918:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504338176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763357.374958:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504338048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763357.375010:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504338176:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763358.014271:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763359.038248:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763360.062240:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763360.446313:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:0.0:1789763360.446358:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763360.446375:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763360.446410:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763360.446418:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763360.446424:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763360.446446:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763360.446455:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789763360.446519:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763357/real 1789763360] req@ffff9018d0f24e00 x1876699504338176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763360.446551:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504338176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763360.446567:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763360.446572:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763360.446576:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504338176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763360.446586:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763357/real 1789763360] req@ffff9018d0f27480 x1876699504337920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763360.446598:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504337920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763360.446604:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763360.446607:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763360.446610:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504337920:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763360.446615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763357/real 1789763360] req@ffff9018d0f25500 x1876699504338048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763360.446623:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504338048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763360.446629:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763360.446631:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763360.446634:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504338048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763360.446638:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763357/real 1789763360] req@ffff9018d0f24700 x1876699504337664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763360.446651:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504337664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763360.446661:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763360.446665:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763360.446670:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504337664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763360.446679:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763357/real 1789763360] req@ffff9018d0f26680 x1876699504337792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763360.446695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504337792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763360.446706:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763360.446710:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763360.446715:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504337792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763360.446735:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763357/real 1789763360] req@ffff9018d0f24000 x1876699504337536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763360.446753:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504337536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763360.446763:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763360.446767:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763360.446770:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504337536:192.168.203.158@tcp:250:kworker.0 00010000:00010000:0.0:1789763361.086172:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763362.110257:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763362.494224:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763362.494237:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763362.494242:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763362.494254:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3289 00000100:00080000:0.0:1789763362.494268:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763362.494277:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763362.494318:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504338304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763362.494355:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763362.494360:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763362.494363:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763362.494375:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3289 00000100:00080000:0.0:1789763362.494380:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763362.494385:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763362.494398:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763362.494436:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504338432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763362.494456:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763362.494461:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763362.494464:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763362.494473:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3289 00000100:00100000:1.0:1789763362.494476:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504338304:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763362.494480:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763362.494496:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763362.494517:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763362.494554:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504338560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763362.494576:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763362.494586:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1789763362.494588:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763362.494594:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763362.494608:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3289 00000100:00080000:0.0:1789763362.494618:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763362.494625:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763362.494634:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763362.494648:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763362/real 1789763362] req@ffff9018c8b99c00 x1876699504338304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763362.494660:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504338688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763362.494694:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763362.494702:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763362.494705:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763362.494712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504338304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763362.494721:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3289 00000100:00080000:0.0:1789763362.494738:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763362.494746:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763362.494751:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763362.494767:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763362.494778:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763362.494799:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504338816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763362.494800:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504338304:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763362.494829:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504338432:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763362.494838:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763362.494849:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763362.494853:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763362.494863:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3289 00000100:00080000:0.0:1789763362.494888:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763362.494891:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763362.494906:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763362.494918:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763362.494929:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763362/real 1789763362] req@ffff9018c8b99500 x1876699504338432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763362.494951:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504338944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763362.494967:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504338432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763362.494996:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763362.495004:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763362.495013:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504338432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763362.495127:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504338560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763362.495175:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763362.495199:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763362/real 1789763362] req@ffff9018c8b98380 x1876699504338560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763362.495225:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504338560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763362.495243:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763362.495249:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763362.495256:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504338560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763362.495269:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504338688:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763362.495311:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763362.495336:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763362/real 1789763362] req@ffff9018c8b9a300 x1876699504338688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763362.495357:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504338688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763362.495372:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763362.495378:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763362.495383:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504338688:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763362.495394:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504338816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763362.495459:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763362.495481:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763362/real 1789763362] req@ffff9018c8b9ad80 x1876699504338816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763362.495505:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504338816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763362.495521:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763362.495527:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763362.495534:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504338816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763362.495546:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504338944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763362.495586:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763362.495608:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763362/real 1789763362] req@ffff9018c8b9a680 x1876699504338944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763362.495648:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504338944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763362.495667:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763362.495673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763362.495680:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504338944:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763363.134291:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763364.158322:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763365.182223:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763366.206168:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763367.230285:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763367.614136:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763367.614146:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763367.614152:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763367.614164:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3294 00000100:00080000:0.0:1789763367.614175:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763367.614183:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763367.614226:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504339072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763367.614255:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763367.614259:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763367.614261:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763367.614265:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3294 00000100:00080000:0.0:1789763367.614269:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763367.614272:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763367.614282:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763367.614295:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504339200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763367.614306:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763367.614311:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763367.614312:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763367.614316:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3294 00000100:00080000:0.0:1789763367.614319:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763367.614326:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763367.614328:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504339072:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789763367.614332:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763367.614348:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504339328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763367.614367:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763367.614371:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763367.614374:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763367.614399:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3294 00000100:00080000:0.0:1789763367.614411:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763367.614416:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763367.614422:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763367.614430:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763367.614438:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504339456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763367.614451:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763367.614455:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763367.614458:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763367.614462:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3294 00000100:00080000:0.0:1789763367.614466:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763367.614471:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763367.614477:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763367.614484:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763367/real 1789763367] req@ffff9018c8b98e00 x1876699504339072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763367.614495:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504339584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763367.614510:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763367.614514:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504339072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763367.614516:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763367.614521:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763367.614533:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3294 00000100:00080000:0.0:1789763367.614540:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763367.614540:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763367.614547:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763367.614552:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:0.0:1789763367.614555:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763367.614561:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504339072:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763367.614571:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504339712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763367.614576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504339200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763367.614617:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763367.614633:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763367/real 1789763367] req@ffff9018c8b9aa00 x1876699504339200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763367.614646:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504339200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763367.614661:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763367.614665:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763367.614668:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504339200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763367.614705:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504339328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763367.614725:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763367.614735:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763367/real 1789763367] req@ffff9018c8b98000 x1876699504339328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763367.614777:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504339328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763367.614794:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763367.614800:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763367.614807:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504339328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763367.614818:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504339456:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763367.614870:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763367.614888:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763367/real 1789763367] req@ffff9018c8b9b800 x1876699504339456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763367.614902:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504339456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763367.614910:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763367.614913:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763367.614917:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504339456:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763367.614923:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504339584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763367.614944:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763367.614955:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763367/real 1789763367] req@ffff9018c8b99880 x1876699504339584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763367.614966:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504339584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763367.614973:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763367.614976:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763367.614979:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504339584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763367.614984:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504339712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763367.614999:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763367.615007:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763367/real 1789763367] req@ffff9018c8b98a80 x1876699504339712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763367.615018:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504339712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763367.615026:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763367.615028:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763367.615031:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504339712:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763368.254229:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763369.278295:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763370.302160:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763371.326277:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763372.350221:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763372.734125:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763372.734135:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763372.734139:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763372.734149:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3299 00000100:00080000:2.0:1789763372.734160:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763372.734168:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763372.734210:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504339840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763372.734233:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763372.734236:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763372.734238:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763372.734241:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3299 00000100:00080000:2.0:1789763372.734245:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763372.734248:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763372.734258:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763372.734265:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504339840:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763372.734271:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504339968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763372.734287:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763372.734289:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763372.734291:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763372.734294:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3299 00000100:00080000:2.0:1789763372.734296:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763372.734299:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763372.734302:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763372.734310:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504340096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763372.734317:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763372.734319:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763372.734321:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763372.734326:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763372.734328:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3299 00000100:00080000:2.0:1789763372.734332:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763372.734335:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763372.734354:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763372.734365:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504340224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763372.734373:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763372.734375:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763372.734377:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763372.734381:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3299 00000100:00080000:2.0:1789763372.734384:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763372.734385:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763372/real 1789763372] req@ffff9018d0f27800 x1876699504339840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763372.734390:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763372.734398:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763372.734408:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504339840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763372.734411:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504340352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763372.734434:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763372.734434:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763372.734441:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763372.734443:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763372.734444:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763372.734449:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504339840:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763372.734450:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3299 00000100:00080000:2.0:1789763372.734457:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763372.734460:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763372.734463:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763372.734470:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504340480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763372.734477:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504339968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763372.734499:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763372.734510:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763372/real 1789763372] req@ffff9018d0f25c00 x1876699504339968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763372.734519:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504339968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763372.734525:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763372.734527:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763372.734529:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504339968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763372.734533:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504340096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763372.734544:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763372.734549:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763372/real 1789763372] req@ffff9018d0f26a00 x1876699504340096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763372.734556:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504340096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763372.734566:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763372.734568:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763372.734570:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504340096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763372.734573:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504340224:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763372.734583:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763372.734588:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763372/real 1789763372] req@ffff9018d0f24a80 x1876699504340224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763372.734595:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504340224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763372.734600:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763372.734602:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763372.734604:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504340224:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763372.734607:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504340352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763372.734616:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763372.734621:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763372/real 1789763372] req@ffff9018d0f24380 x1876699504340352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763372.734627:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504340352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763372.734633:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763372.734635:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763372.734637:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504340352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763372.734647:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504340480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763372.734657:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763372.734663:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763372/real 1789763372] req@ffff9018d0f25180 x1876699504340480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763372.734670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504340480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763372.734675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763372.734676:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763372.734678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504340480:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763373.374236:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763374.398242:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763375.422293:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763376.446242:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763377.470330:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763377.854217:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763377.854230:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763377.854235:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763377.854248:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3304 00000100:00080000:0.0:1789763377.854262:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763377.854271:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763377.854389:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504340608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763377.854426:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763377.854432:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763377.854434:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763377.854439:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3304 00000100:00080000:0.0:1789763377.854445:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763377.854450:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763377.854463:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763377.854479:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504340736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763377.854493:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763377.854497:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763377.854499:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763377.854503:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3304 00000100:00080000:0.0:1789763377.854506:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763377.854511:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763377.854517:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763377.854529:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504340864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763377.854540:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763377.854545:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763377.854548:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763377.854553:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3304 00000100:00080000:0.0:1789763377.854556:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763377.854565:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763377.854568:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504340608:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789763377.854573:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763377.854590:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504340992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763377.854605:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763377.854609:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763377.854611:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763377.854616:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3304 00000100:00080000:0.0:1789763377.854622:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763377.854627:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763377.854634:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763377.854648:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504341120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763377.854659:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763377.854667:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1789763377.854669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763377.854671:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763377.854683:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3304 00000100:00080000:0.0:1789763377.854689:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763377.854694:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763377.854701:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763377.854719:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504341248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763377.854755:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763377/real 1789763377] req@ffff9018c8b99f80 x1876699504340608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763377.854786:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504340608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763377.854801:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763377.854806:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763377.854812:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504340608:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763377.854821:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504340736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763377.854849:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763377.854865:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763377/real 1789763377] req@ffff9018c8b98700 x1876699504340736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763377.854881:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504340736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763377.854891:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763377.854894:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763377.854898:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504340736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763377.854906:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504340864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763377.854925:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763377.854936:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763377/real 1789763377] req@ffff9018c8b9bb80 x1876699504340864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763377.854950:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504340864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763377.854960:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763377.854963:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763377.854967:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504340864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763377.854997:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504340992:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763377.855018:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763377.855029:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763377/real 1789763377] req@ffff9018c8b9b100 x1876699504340992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763377.855087:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504340992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763377.855098:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763377.855102:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763377.855106:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504340992:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763377.855113:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504341120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763377.855138:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763377.855151:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763377/real 1789763377] req@ffff9018c8b9b480 x1876699504341120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763377.855165:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504341120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763377.855175:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763377.855178:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763377.855182:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504341120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763377.855189:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504341248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763377.855207:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763377.855218:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763377/real 1789763377] req@ffff9018c8b98a80 x1876699504341248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763377.855232:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504341248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763377.855241:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763377.855244:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763377.855248:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504341248:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763378.494217:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763379.518273:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763380.542325:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763381.566260:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763382.590229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763382.974218:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763382.974239:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763382.974248:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763382.974297:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3309 00000100:00080000:2.0:1789763382.974319:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763382.974334:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763382.974407:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504341376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763382.974453:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763382.974458:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763382.974461:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763382.974468:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3309 00000100:00080000:2.0:1789763382.974473:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763382.974479:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763382.974493:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763382.974512:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504341504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763382.974528:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763382.974533:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763382.974536:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763382.974541:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3309 00000100:00080000:2.0:1789763382.974545:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763382.974555:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763382.974560:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504341376:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789763382.974565:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763382.974596:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504341632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763382.974620:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763382.974628:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763382.974634:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763382.974643:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3309 00000100:00080000:2.0:1789763382.974654:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763382.974677:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789763382.974684:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789763382.974694:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763382.974721:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504341760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763382.974737:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763382.974742:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763382.974745:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763382.974752:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3309 00000100:00080000:2.0:1789763382.974758:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763382.974760:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763382/real 1789763382] req@ffff9018d0f26300 x1876699504341376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763382.974772:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763382.974787:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763382.974815:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504341888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763382.974826:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504341376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763382.974835:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763382.974851:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763382.974859:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763382.974874:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3309 00000100:00080000:1.0:1789763382.974881:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763382.974881:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763382.974893:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763382.974901:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763382.974902:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763382.974915:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504341376:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763382.974921:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504342016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763382.974936:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504341504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763382.974991:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763382.975015:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763382/real 1789763382] req@ffff9018d0f26d80 x1876699504341504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763382.975082:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504341504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763382.975095:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763382.975100:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763382.975105:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504341504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763382.975145:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504341632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763382.975173:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763382.975188:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763382/real 1789763382] req@ffff9018d0f25f80 x1876699504341632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763382.975203:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504341632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763382.975214:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763382.975217:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763382.975222:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504341632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763382.975228:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504341760:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763382.975249:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763382.975260:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763382/real 1789763382] req@ffff9018d0f27100 x1876699504341760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763382.975312:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504341760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763382.975324:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763382.975328:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763382.975332:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504341760:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763382.975340:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504341888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763382.975369:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763382.975383:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763382/real 1789763382] req@ffff9018d0f25180 x1876699504341888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763382.975397:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504341888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763382.975408:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763382.975412:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763382.975416:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504341888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763382.975423:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504342016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763382.975442:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763382.975452:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763382/real 1789763382] req@ffff9018d0f24380 x1876699504342016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763382.975466:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504342016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763382.975476:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763382.975480:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763382.975483:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504342016:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763383.614192:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763384.638339:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763385.662299:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763386.686206:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763387.710305:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763388.094251:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763388.094269:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763388.094277:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763388.094293:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3314 00000100:00080000:1.0:1789763388.094309:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763388.094321:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763388.094380:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504342144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763388.094436:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763388.094445:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763388.094449:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763388.094459:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3314 00000100:00080000:1.0:1789763388.094469:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763388.094480:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763388.094501:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763388.094519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504342144:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763388.094538:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504342272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763388.094568:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763388.094592:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763388.094596:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763388.094605:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3314 00000100:00080000:1.0:1789763388.094618:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763388.094624:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763388.094633:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763388.094651:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504342400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763388.094659:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763388.094678:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763388.094701:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763388.094706:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763388.094713:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3314 00000100:00080000:1.0:1789763388.094719:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763388.094725:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763388.094737:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763388.094747:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763388/real 1789763388] req@ffff9018c66a4e00 x1876699504342144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763388.094763:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504342528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763388.094787:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504342144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763388.094797:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763388.094827:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763388.094828:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763388.094838:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763388.094849:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789763388.094861:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3314 00000100:00100000:2.0:1789763388.094874:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504342144:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763388.094877:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763388.094910:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763388.094923:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763388.094955:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504342272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763388.094979:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504342656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763388.095002:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763388.095015:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763388.095026:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789763388.095075:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763388/real 1789763388] req@ffff9018c66a5500 x1876699504342272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763388.095100:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504342272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763388.095111:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763388.095116:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763388.095143:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3314 00000100:00080000:2.0:1789763388.095143:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763388.095165:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504342272:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789763388.095169:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763388.095180:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504342400:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789763388.095189:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763388.095207:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763388.095256:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763388.095280:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504342784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763388.095288:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763388/real 1789763388] req@ffff9018c66a6680 x1876699504342400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763388.095406:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504342400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763388.095423:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763388.095429:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763388.095434:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504342400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763388.095446:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504342528:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763388.095477:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763388.095490:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763388/real 1789763388] req@ffff9018c66a4700 x1876699504342528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763388.095505:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504342528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763388.095515:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763388.095520:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763388.095524:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504342528:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763388.095550:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504342656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763388.095595:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763388.095616:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763388/real 1789763388] req@ffff9018c66a6300 x1876699504342656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763388.095663:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504342656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763388.095680:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763388.095687:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763388.095693:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504342656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763388.095705:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504342784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763388.095747:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763388.095771:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763388/real 1789763388] req@ffff9018c66a5180 x1876699504342784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763388.095798:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504342784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763388.095825:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763388.095831:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763388.095839:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504342784:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763388.734327:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763389.758311:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763390.782249:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763391.806287:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763392.830178:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763393.214169:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763393.214175:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763393.214179:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763393.214188:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3319 00000100:00080000:0.0:1789763393.214210:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763393.214216:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763393.214251:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504342912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763393.214277:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763393.214281:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763393.214282:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763393.214286:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3319 00000100:00080000:0.0:1789763393.214289:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763393.214292:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763393.214302:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763393.214313:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504343040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763393.214321:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504342912:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763393.214326:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763393.214330:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763393.214331:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763393.214334:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3319 00000100:00080000:0.0:1789763393.214337:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763393.214339:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763393.214343:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763393.214352:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504343168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763393.214359:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763393.214362:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763393.214363:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763393.214366:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3319 00000100:00080000:0.0:1789763393.214370:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763393.214372:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763393.214377:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763393.214381:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763393.214386:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504343296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763393.214395:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763393.214397:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763393.214400:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763393.214405:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3319 00000100:00080000:0.0:1789763393.214407:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763393.214411:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763393.214416:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763393.214430:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763393/real 1789763393] req@ffff9018c8b99880 x1876699504342912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763393.214431:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504343424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763393.214465:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763393.214470:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763393.214471:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763393.214478:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3319 00000100:00100000:1.0:1789763393.214478:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504342912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763393.214484:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763393.214491:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763393.214495:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789763393.214496:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763393.214505:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763393.214510:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504343552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763393.214514:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504342912:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763393.214551:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504343040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763393.214579:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763393.214591:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763393/real 1789763393] req@ffff9018c8b9b800 x1876699504343040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763393.214601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504343040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763393.214609:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763393.214612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763393.214616:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504343040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763393.214622:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504343168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763393.214640:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763393.214649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763393/real 1789763393] req@ffff9018c8b98000 x1876699504343168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763393.214659:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504343168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763393.214672:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763393.214676:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763393.214679:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504343168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763393.214688:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504343296:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763393.214711:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763393.214723:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763393/real 1789763393] req@ffff9018c8b9aa00 x1876699504343296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763393.214736:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504343296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763393.214742:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763393.214745:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763393.214747:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504343296:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763393.214754:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504343424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763393.214764:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763393.214771:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763393/real 1789763393] req@ffff9018c8b98e00 x1876699504343424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763393.214778:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504343424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763393.214783:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763393.214784:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763393.214787:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504343424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763393.214790:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504343552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763393.214801:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763393.214806:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763393/real 1789763393] req@ffff9018c8b9a680 x1876699504343552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763393.214813:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504343552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763393.214818:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763393.214820:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763393.214822:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504343552:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763393.854225:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763394.878250:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763395.902301:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763396.926235:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763397.950306:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763398.334219:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763398.334233:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763398.334239:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763398.334252:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3324 00000100:00080000:3.0:1789763398.334277:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763398.334287:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763398.334335:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504343680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763398.334372:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763398.334377:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763398.334380:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763398.334385:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3324 00000100:00080000:3.0:1789763398.334389:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763398.334395:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763398.334408:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763398.334426:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504343808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763398.334443:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504343680:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763398.334446:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763398.334460:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763398.334463:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763398.334468:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3324 00000100:00080000:3.0:1789763398.334472:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763398.334476:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763398.334484:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763398.334514:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504343936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763398.334528:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763398.334534:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763398.334537:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763398.334542:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3324 00000100:00080000:3.0:1789763398.334546:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763398.334560:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763398.334569:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763398.334570:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763398.334593:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504344064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763398.334608:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763398.334613:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763398.334615:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763398.334621:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3324 00000100:00080000:3.0:1789763398.334627:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763398.334632:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763398.334640:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763398.334642:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763398/real 1789763398] req@ffff9018d070e300 x1876699504343680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763398.334659:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504344192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763398.334675:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763398.334680:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504343680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763398.334681:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763398.334688:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763398.334705:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3324 00000100:00080000:3.0:1789763398.334715:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763398.334725:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763398.334729:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789763398.334732:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763398.334742:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763398.334756:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504343680:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763398.334757:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504344320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763398.334796:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504343808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763398.334844:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763398.334862:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763398/real 1789763398] req@ffff9018d070c000 x1876699504343808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763398.334878:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504343808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763398.334889:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763398.334904:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763398.334910:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504343808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763398.334921:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504343936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763398.334957:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763398.334979:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763398/real 1789763398] req@ffff9018d070dc00 x1876699504343936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763398.335005:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504343936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763398.335021:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763398.335027:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763398.335076:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504343936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763398.335086:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504344064:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763398.335118:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763398.335134:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763398/real 1789763398] req@ffff9018d070d500 x1876699504344064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763398.335149:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504344064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763398.335198:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763398.335203:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763398.335207:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504344064:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763398.335215:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504344192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763398.335244:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763398.335267:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763398/real 1789763398] req@ffff9018d070f800 x1876699504344192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763398.335281:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504344192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763398.335291:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763398.335295:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763398.335299:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504344192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763398.335305:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504344320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763398.335324:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763398.335334:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763398/real 1789763398] req@ffff9018d070f100 x1876699504344320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763398.335348:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504344320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763398.335357:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763398.335360:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763398.335365:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504344320:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763398.974245:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763399.998266:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763401.022295:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763402.046219:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763403.070202:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763403.454109:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763403.454132:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763403.454136:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763403.454148:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3330 00000100:00080000:2.0:1789763403.454159:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763403.454167:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763403.454210:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504344448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763403.454232:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763403.454236:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763403.454238:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763403.454243:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3330 00000100:00080000:2.0:1789763403.454247:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763403.454251:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763403.454262:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763403.454276:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504344576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763403.454277:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504344448:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763403.454288:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763403.454291:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763403.454293:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763403.454297:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3330 00000100:00080000:2.0:1789763403.454300:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763403.454304:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763403.454308:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763403.454317:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504344704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763403.454333:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763403.454336:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763403.454338:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763403.454339:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763403.454342:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3330 00000100:00080000:2.0:1789763403.454346:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763403.454350:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763403.454355:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763403.454365:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504344832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763403.454374:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763403.454378:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763403.454380:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763403.454384:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3330 00000100:00080000:2.0:1789763403.454388:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763403.454389:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763403/real 1789763403] req@ffff9018d0f24a80 x1876699504344448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763403.454392:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763403.454397:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763403.454417:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504344960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763403.454424:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504344448/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763403.454428:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763403.454432:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763403.454435:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763403.454436:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763403.454439:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3330 00000100:00080000:1.0:1789763403.454441:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763403.454443:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763403.454445:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504344448:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763403.454447:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763403.454451:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763403.454460:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504345088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763403.454469:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504344576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763403.454493:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763403.454508:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763403/real 1789763403] req@ffff9018d0f26a00 x1876699504344576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763403.454522:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504344576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763403.454531:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763403.454534:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763403.454537:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504344576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763403.454541:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504344704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763403.454558:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763403.454566:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763403/real 1789763403] req@ffff9018d0f25c00 x1876699504344704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763403.454592:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504344704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763403.454598:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763403.454600:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763403.454603:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504344704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763403.454607:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504344832:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763403.454620:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763403.454627:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763403/real 1789763403] req@ffff9018d0f27800 x1876699504344832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763403.454635:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504344832/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763403.454641:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763403.454643:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763403.454645:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504344832:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763403.454649:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504344960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763403.454659:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763403.454665:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763403/real 1789763403] req@ffff9018d0f24000 x1876699504344960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763403.454673:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504344960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763403.454678:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763403.454681:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763403.454683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504344960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763403.454696:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504345088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763403.454707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763403.454713:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763403/real 1789763403] req@ffff9018d0f26680 x1876699504345088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763403.454721:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504345088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763403.454726:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763403.454728:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763403.454730:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504345088:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763404.094241:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763405.118220:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763406.142326:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763407.166299:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763408.190320:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763408.574169:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763408.574176:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763408.574180:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763408.574187:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3335 00000100:00080000:0.0:1789763408.574196:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763408.574202:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763408.574233:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504345216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763408.574254:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763408.574257:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763408.574258:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763408.574261:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3335 00000100:00080000:0.0:1789763408.574263:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763408.574265:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763408.574272:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763408.574281:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504345344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763408.574288:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763408.574290:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763408.574292:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763408.574295:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3335 00000100:00080000:0.0:1789763408.574297:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763408.574300:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763408.574303:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763408.574309:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504345472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763408.574316:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763408.574318:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763408.574320:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763408.574320:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504345216:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763408.574326:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3335 00000100:00080000:0.0:1789763408.574332:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763408.574334:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763408.574338:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763408.574347:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504345600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763408.574353:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763408.574356:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763408.574357:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763408.574359:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3335 00000100:00080000:0.0:1789763408.574361:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763408.574363:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763408.574366:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763408.574373:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504345728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763408.574379:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763408.574382:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763408.574383:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763408.574385:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3335 00000100:00080000:0.0:1789763408.574387:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763408.574392:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763408.574393:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763408.574399:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763408.574408:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504345856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763408.574438:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763408/real 1789763408] req@ffff9018c8b9ad80 x1876699504345216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763408.574453:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504345216/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763408.574461:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763408.574464:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763408.574468:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504345216:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763408.574473:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504345344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763408.574491:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763408.574507:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763408/real 1789763408] req@ffff9018c8b9a300 x1876699504345344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763408.574519:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504345344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763408.574528:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763408.574531:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763408.574534:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504345344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763408.574557:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504345472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763408.574578:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763408.574590:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763408/real 1789763408] req@ffff9018c8b98380 x1876699504345472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763408.574602:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504345472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763408.574609:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763408.574612:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763408.574614:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504345472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763408.574618:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504345600:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763408.574633:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763408.574641:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763408/real 1789763408] req@ffff9018c8b99500 x1876699504345600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763408.574648:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504345600/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763408.574654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763408.574655:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763408.574657:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504345600:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763408.574661:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504345728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763408.574671:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763408.574677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763408/real 1789763408] req@ffff9018c8b99c00 x1876699504345728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763408.574684:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504345728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763408.574689:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763408.574690:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763408.574692:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504345728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763408.574695:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504345856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763408.574704:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763408.574710:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763408/real 1789763408] req@ffff9018c8b9a680 x1876699504345856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763408.574716:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504345856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763408.574723:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763408.574725:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763408.574728:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504345856:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763409.214256:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763410.238301:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763411.262286:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763412.286273:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763413.310311:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763413.694220:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763413.694232:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763413.694238:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763413.694251:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3340 00000100:00080000:0.0:1789763413.694265:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763413.694274:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763413.694327:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504345984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763413.694379:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763413.694387:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763413.694391:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763413.694401:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3340 00000100:00080000:0.0:1789763413.694410:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763413.694420:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763413.694438:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763413.694465:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504346112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763413.694478:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504345984:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763413.694486:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763413.694502:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763413.694509:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763413.694516:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3340 00000100:00080000:0.0:1789763413.694521:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763413.694526:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763413.694535:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763413.694551:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504346240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763413.694563:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763413.694568:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763413.694572:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763413.694576:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3340 00000100:00080000:0.0:1789763413.694599:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763413.694605:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763413.694612:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763413.694628:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504346368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763413.694642:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763413.694659:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763413.694666:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763413.694668:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763413.694674:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3340 00000100:00080000:0.0:1789763413.694680:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763413.694686:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763413.694693:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763413.694708:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504346496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763413.694720:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763413.694724:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763413.694727:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763413.694731:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3340 00000100:00080000:0.0:1789763413.694737:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763413.694741:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763413.694746:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763413.694760:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504346624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763413.694771:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763413/real 1789763413] req@ffff9018c8b98e00 x1876699504345984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763413.694805:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504345984/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763413.694823:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763413.694833:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763413.694841:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504345984:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763413.694887:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504346112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763413.694926:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763413.694942:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763413/real 1789763413] req@ffff9018c8b9aa00 x1876699504346112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763413.694971:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504346112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763413.694983:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763413.694987:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763413.694991:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504346112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763413.695000:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504346240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763413.695024:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763413.695130:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763413/real 1789763413] req@ffff9018c8b98000 x1876699504346240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763413.695155:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504346240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763413.695168:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763413.695173:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763413.695178:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504346240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763413.695187:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504346368:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763413.695220:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763413.695252:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763413/real 1789763413] req@ffff9018c8b9b800 x1876699504346368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763413.695266:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504346368/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763413.695276:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763413.695280:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763413.695283:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504346368:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763413.695290:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504346496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763413.695313:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763413.695324:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763413/real 1789763413] req@ffff9018c8b99880 x1876699504346496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763413.695338:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504346496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763413.695347:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763413.695350:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763413.695354:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504346496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763413.695360:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504346624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763413.695378:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763413.695389:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763413/real 1789763413] req@ffff9018c8b98a80 x1876699504346624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763413.695402:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504346624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763413.695411:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763413.695414:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763413.695418:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504346624:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763414.334303:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763415.358253:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763416.382247:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763417.406262:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763418.430271:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763418.814178:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763418.814191:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763418.814196:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763418.814208:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3345 00000100:00080000:0.0:1789763418.814222:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763418.814232:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763418.814292:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504346752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763418.814352:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763418.814360:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763418.814364:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763418.814373:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3345 00000100:00080000:0.0:1789763418.814380:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763418.814389:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763418.814407:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763418.814436:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504346880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763418.814446:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504346752:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763418.814474:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763418.814479:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763418.814482:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763418.814489:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3345 00000100:00080000:0.0:1789763418.814495:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763418.814500:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763418.814516:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763418.814536:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504347008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763418.814549:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763418.814555:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763418.814558:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763418.814562:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3345 00000100:00080000:0.0:1789763418.814569:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763418.814576:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763418.814583:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763418.814593:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763418.814609:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504347136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763418.814622:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763418.814626:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763418.814628:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763418.814632:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3345 00000100:00080000:0.0:1789763418.814636:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763418.814641:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763418.814647:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763418.814654:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763418/real 1789763418] req@ffff9018c8b9b480 x1876699504346752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763418.814665:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504347264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763418.814692:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763418.814697:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763418.814704:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763418.814715:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3345 00000100:00080000:0.0:1789763418.814722:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763418.814727:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763418.814734:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763418.814745:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504346752/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763418.814752:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504347392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763418.814830:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763418.814839:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763418.814846:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504346752:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763418.814877:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504346880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763418.814914:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763418.814932:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763418/real 1789763418] req@ffff9018c8b9b100 x1876699504346880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763418.814948:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504346880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763418.814959:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763418.814963:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763418.814967:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504346880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763418.814974:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504347008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763418.814994:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763418.815077:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763418/real 1789763418] req@ffff9018c8b9bb80 x1876699504347008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763418.815095:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504347008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763418.815106:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763418.815110:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763418.815114:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504347008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763418.815123:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504347136:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763418.815152:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763418.815166:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763418/real 1789763418] req@ffff9018c8b98700 x1876699504347136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763418.815180:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504347136/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763418.815190:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763418.815193:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763418.815197:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504347136:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763418.815204:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504347264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763418.815223:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763418.815235:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763418/real 1789763418] req@ffff9018c8b99f80 x1876699504347264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763418.815259:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504347264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763418.815269:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763418.815272:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763418.815275:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504347264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763418.815282:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504347392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763418.815300:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763418.815311:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763418/real 1789763418] req@ffff9018c8b98a80 x1876699504347392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763418.815325:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504347392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763418.815335:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763418.815338:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763418.815341:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504347392:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763419.454276:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763420.478253:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763421.502269:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763422.526312:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763423.550267:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763423.934161:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763423.934174:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763423.934179:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763423.934193:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3350 00000100:00080000:0.0:1789763423.934206:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763423.934216:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763423.934289:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504347520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763423.934325:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763423.934331:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763423.934334:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763423.934340:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3350 00000100:00080000:0.0:1789763423.934346:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763423.934352:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763423.934367:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763423.934391:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504347648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763423.934396:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504347520:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763423.934423:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763423.934427:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763423.934466:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763423.934474:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3350 00000100:00080000:0.0:1789763423.934506:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763423.934512:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763423.934520:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763423.934541:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504347776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763423.934556:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763423.934560:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763423.934564:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763423.934575:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3350 00000100:00100000:1.0:1789763423.934579:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504347648:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763423.934585:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763423.934601:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763423.934627:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763423.934648:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504347904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763423.934665:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504347776:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763423.934674:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763423.934682:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763423.934685:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763423.934691:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3350 00000100:00080000:0.0:1789763423.934698:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763423.934703:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763423.934728:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763423.934752:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504347904:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763423.934754:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504348032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763423.934777:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763423.934781:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763423.934785:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763423.934792:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3350 00000100:00080000:0.0:1789763423.934798:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763423.934804:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763423.934811:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763423.934817:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504348032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763423.934833:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504348160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763423.934885:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504348160:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763424.574189:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763425.598242:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763426.622274:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763427.006374:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789763427.006433:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763427.006453:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763427.006508:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763427.006527:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763427.006543:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763427.006557:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763427.006588:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789763427.006661:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763423/real 1789763427] req@ffff9018c8b9a680 x1876699504348160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763427.006709:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504348160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763427.006732:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763427.006741:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763427.006750:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504348160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763427.006767:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763423/real 1789763427] req@ffff9018c8b98e00 x1876699504348032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763427.006794:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504348032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763427.006811:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763427.006818:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763427.006825:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504348032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763427.006838:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763423/real 1789763427] req@ffff9018c8b9aa00 x1876699504347904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763427.006860:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504347904/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763427.006876:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763427.006882:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763427.006889:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504347904:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763427.006901:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763423/real 1789763427] req@ffff9018c8b9b800 x1876699504347648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763427.006927:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504347648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763427.006974:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763427.006982:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763427.006989:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504347648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763427.007002:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763423/real 1789763427] req@ffff9018c8b98000 x1876699504347776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763427.007026:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504347776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763427.007091:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763427.007099:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763427.007106:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504347776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763427.007131:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763423/real 1789763427] req@ffff9018c8b99880 x1876699504347520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763427.007158:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504347520/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763427.007174:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763427.007180:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763427.007187:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504347520:192.168.203.158@tcp:250:kworker.0 00010000:00010000:2.0:1789763427.646226:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763428.670295:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763429.054213:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763429.054231:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763429.054238:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763429.054256:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3355 00000100:00080000:0.0:1789763429.054274:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763429.054288:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763429.054374:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504348288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763429.054424:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763429.054434:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763429.054440:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763429.054455:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3355 00000100:00080000:0.0:1789763429.054466:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763429.054477:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763429.054501:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763429.054502:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504348288:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763429.054552:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504348416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763429.054582:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763429.054591:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763429.054596:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763429.054607:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3355 00000100:00080000:0.0:1789763429.054622:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763429.054635:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763429.054645:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763429.054663:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763429.054698:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504348544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763429.054719:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763429.054727:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763429.054731:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763429.054745:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763429/real 1789763429] req@ffff9018c8b99c00 x1876699504348288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763429.054747:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3355 00000100:00080000:0.0:1789763429.054762:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763429.054784:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763429.054804:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763429.054828:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504348288/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763429.054839:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504348672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763429.054960:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763429.054965:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763429.054984:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763429.054990:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789763429.054995:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763429.055013:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504348288:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763429.055016:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3355 00000100:00080000:0.0:1789763429.055100:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763429.055116:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763429.055133:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763429.055164:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504348800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763429.055175:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504348416:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763429.055215:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763429.055225:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763429.055229:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763429.055240:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3355 00000100:00080000:0.0:1789763429.055252:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763429.055261:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789763429.055279:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789763429.055286:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763429.055322:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504348928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763429.055328:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763429/real 1789763429] req@ffff9018c8b99500 x1876699504348416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763429.055416:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504348416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763429.055438:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763429.055448:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763429.055456:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504348416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763429.055472:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504348544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763429.055520:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763429.055545:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763429/real 1789763429] req@ffff9018c8b98380 x1876699504348544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763429.055568:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504348544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763429.055583:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763429.055589:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763429.055595:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504348544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763429.055607:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504348672:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763429.055649:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763429.055672:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763429/real 1789763429] req@ffff9018c8b9a300 x1876699504348672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763429.055696:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504348672/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763429.055711:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763429.055717:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763429.055723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504348672:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763429.055758:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504348800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763429.055801:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763429.055841:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763429/real 1789763429] req@ffff9018c8b9ad80 x1876699504348800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763429.055869:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504348800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763429.055885:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763429.055892:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763429.055898:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504348800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763429.055911:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504348928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763429.055979:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763429.056003:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763429/real 1789763429] req@ffff9018c8b99c00 x1876699504348928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763429.056098:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504348928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763429.056116:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763429.056122:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763429.056129:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504348928:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763429.694245:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763430.718282:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763431.742229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763432.766262:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763433.790214:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763434.174175:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763434.174189:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763434.174194:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763434.174207:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3360 00000100:00080000:1.0:1789763434.174221:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763434.174230:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763434.174277:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504349056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763434.174317:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763434.174322:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763434.174324:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763434.174330:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3360 00000100:00080000:1.0:1789763434.174334:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763434.174339:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763434.174352:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763434.174369:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504349184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763434.174397:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504349056:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763434.174431:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763434.174439:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763434.174444:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763434.174454:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3360 00000100:00080000:1.0:1789763434.174465:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763434.174474:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763434.174484:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763434.174493:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763434.174513:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504349312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763434.174529:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763434.174534:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763434.174537:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763434.174543:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3360 00000100:00080000:1.0:1789763434.174550:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763434.174551:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763434/real 1789763434] req@ffff9018c66a7800 x1876699504349056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763434.174563:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763434.174578:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763434.174599:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504349440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763434.174612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504349056/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763434.174618:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763434.174631:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763434.174637:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763434.174647:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763434.174648:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3360 00000100:00080000:1.0:1789763434.174659:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763434.174660:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789763434.174671:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763434.174676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504349056:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789763434.174698:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763434.174712:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504349184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763434.174721:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504349568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763434.174740:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763434.174745:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1789763434.174753:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763434.174761:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763434.174770:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3360 00000100:00080000:1.0:1789763434.174777:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763434.174778:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763434/real 1789763434] req@ffff9018c66a7b80 x1876699504349184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763434.174791:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763434.174805:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763434.174817:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504349184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763434.174829:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504349696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763434.174837:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763434.174851:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763434.174864:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504349184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763434.174879:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504349312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763434.174940:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763434.174956:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763434/real 1789763434] req@ffff9018c66a4000 x1876699504349312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763434.174972:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504349312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763434.174990:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763434.174995:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763434.174999:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504349312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763434.175006:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504349440:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763434.175026:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763434.175080:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763434/real 1789763434] req@ffff9018c66a5f80 x1876699504349440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763434.175095:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504349440/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763434.175112:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763434.175119:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763434.175125:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504349440:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763434.175160:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504349568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763434.175202:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763434.175225:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763434/real 1789763434] req@ffff9018c66a6d80 x1876699504349568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763434.175242:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504349568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763434.175253:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763434.175257:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763434.175262:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504349568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763434.175270:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504349696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763434.175295:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763434.175307:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763434/real 1789763434] req@ffff9018c66a5c00 x1876699504349696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763434.175321:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504349696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763434.175331:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763434.175334:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763434.175337:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504349696:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763434.814292:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763435.838229:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763436.862228:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763437.886245:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763438.910261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763439.294185:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763439.294196:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763439.294201:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763439.294212:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3365 00000100:00080000:2.0:1789763439.294224:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763439.294233:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763439.294275:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504349824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763439.294310:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763439.294314:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763439.294316:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763439.294320:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3365 00000100:00080000:2.0:1789763439.294324:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763439.294330:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763439.294341:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763439.294356:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504349952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763439.294363:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504349824:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763439.294378:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763439.294383:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763439.294385:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763439.294390:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3365 00000100:00080000:2.0:1789763439.294393:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763439.294397:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763439.294402:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763439.294418:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504350080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763439.294427:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763439.294431:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763439.294434:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763439.294438:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3365 00000100:00080000:2.0:1789763439.294442:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763439.294446:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789763439.294453:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789763439.294455:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763439.294472:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504350208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763439.294491:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763439.294496:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763439.294498:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763439.294501:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3365 00000100:00080000:2.0:1789763439.294504:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763439.294508:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763439.294514:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763439.294517:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763439/real 1789763439] req@ffff9018d0f24700 x1876699504349824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763439.294531:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504350336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763439.294543:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763439.294546:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504349824/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763439.294548:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763439.294554:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763439.294569:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3365 00000100:00080000:1.0:1789763439.294575:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763439.294576:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763439.294588:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763439.294590:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:2.0:1789763439.294595:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763439.294599:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504349824:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763439.294613:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504350464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763439.294635:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504349952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763439.294664:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763439.294678:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763439/real 1789763439] req@ffff9018d0f25500 x1876699504349952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763439.294690:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504349952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763439.294698:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763439.294702:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763439.294705:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504349952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763439.294711:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504350080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763439.294726:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763439.294735:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763439/real 1789763439] req@ffff9018d0f27480 x1876699504350080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763439.294746:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504350080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763439.294754:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763439.294757:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763439.294760:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504350080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763439.294765:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504350208:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763439.294781:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763439.294792:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763439/real 1789763439] req@ffff9018d0f24e00 x1876699504350208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763439.294812:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504350208/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763439.294824:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763439.294829:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763439.294834:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504350208:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763439.294844:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504350336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763439.294896:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763439.294915:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763439/real 1789763439] req@ffff9018d0f26680 x1876699504350336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763439.294927:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504350336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763439.294936:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763439.294939:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763439.294942:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504350336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763439.294947:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504350464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763439.294967:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763439.294977:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763439/real 1789763439] req@ffff9018d0f24000 x1876699504350464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763439.294988:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504350464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763439.295003:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763439.295006:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763439.295009:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504350464:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763439.934273:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763440.958246:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763441.982274:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763443.006273:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763444.030285:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763444.414150:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763444.414163:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763444.414169:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763444.414181:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3370 00000100:00080000:2.0:1789763444.414214:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763444.414226:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763444.414305:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504350592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763444.414348:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763444.414353:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763444.414356:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763444.414361:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3370 00000100:00080000:2.0:1789763444.414368:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763444.414373:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763444.414387:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763444.414407:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504350720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763444.414422:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763444.414427:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763444.414429:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763444.414435:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3370 00000100:00080000:2.0:1789763444.414439:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763444.414446:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763444.414452:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763444.414463:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504350848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763444.414473:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504350592:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763444.414527:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763444.414534:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763444.414538:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763444.414546:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3370 00000100:00080000:2.0:1789763444.414554:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763444.414562:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763444.414603:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763444.414620:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763444.414628:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504350976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763444.414646:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763444.414652:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763444.414655:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763444.414659:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3370 00000100:00080000:2.0:1789763444.414666:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763444.414672:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763444.414679:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763444.414692:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504351104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763444.414701:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763444/real 1789763444] req@ffff9018d0f27800 x1876699504350592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763444.414715:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763444.414730:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763444.414735:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763444.414743:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3370 00000100:00080000:2.0:1789763444.414749:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763444.414758:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504350592/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763444.414760:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763444.414771:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763444.414806:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504351232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763444.414810:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763444.414934:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763444.414941:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504350592:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763444.414952:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504350720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763444.414991:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763444.415010:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763444/real 1789763444] req@ffff9018d0f25c00 x1876699504350720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763444.415027:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504350720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763444.415081:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763444.415086:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763444.415091:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504350720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763444.415141:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504350848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763444.415169:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763444.415201:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763444/real 1789763444] req@ffff9018d0f26a00 x1876699504350848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763444.415217:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504350848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763444.415228:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763444.415232:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763444.415236:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504350848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763444.415243:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504350976:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763444.415264:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763444.415276:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763444/real 1789763444] req@ffff9018d0f24a80 x1876699504350976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763444.415299:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504350976/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763444.415315:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763444.415321:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763444.415328:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504350976:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763444.415340:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504351104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763444.415380:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763444.415403:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763444/real 1789763444] req@ffff9018d0f24380 x1876699504351104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763444.415425:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504351104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763444.415436:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763444.415439:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763444.415444:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504351104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763444.415455:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504351232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763444.415494:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763444.415533:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763444/real 1789763444] req@ffff9018d0f25180 x1876699504351232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763444.415556:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504351232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763444.415570:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763444.415593:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763444.415598:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504351232:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763445.054288:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763446.078234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763447.102203:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763448.126270:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763449.150319:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763449.534207:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763449.534224:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763449.534231:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763449.534248:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3376 00000100:00080000:3.0:1789763449.534267:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763449.534279:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763449.534336:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504351360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763449.534376:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763449.534382:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763449.534385:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763449.534390:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3376 00000100:00080000:3.0:1789763449.534396:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763449.534401:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763449.534413:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763449.534429:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504351488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763449.534443:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763449.534448:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763449.534452:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763449.534458:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3376 00000100:00080000:3.0:1789763449.534462:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763449.534467:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763449.534475:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763449.534484:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504351360:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763449.534495:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504351616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763449.534510:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763449.534515:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763449.534518:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763449.534523:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3376 00000100:00080000:3.0:1789763449.534527:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763449.534532:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763449.534539:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763449.534553:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504351744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763449.534566:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763449.534571:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763449.534573:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763449.534583:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763449.534590:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3376 00000100:00080000:3.0:1789763449.534597:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763449.534604:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763449.534611:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763449.534630:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504351872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763449.534641:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763449.534645:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789763449.534652:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763449/real 1789763449] req@ffff9018d070e680 x1876699504351360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763449.534653:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763449.534667:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3376 00000100:00080000:3.0:1789763449.534677:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763449.534689:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763449.534696:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763449.534701:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504351360/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763449.534716:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504352000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763449.534803:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763449.534814:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763449.534822:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504351360:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763449.534837:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504351488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763449.534874:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763449.534892:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763449/real 1789763449] req@ffff9018d070d180 x1876699504351488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763449.534908:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504351488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763449.534919:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763449.534923:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763449.534927:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504351488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763449.534957:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504351616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763449.534978:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763449.534990:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763449/real 1789763449] req@ffff9018d070ca80 x1876699504351616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763449.535004:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504351616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763449.535014:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763449.535017:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763449.535021:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504351616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763449.535029:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504351744:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763449.535101:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763449.535116:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763449/real 1789763449] req@ffff9018d070d880 x1876699504351744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763449.535131:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504351744/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763449.535141:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763449.535145:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763449.535149:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504351744:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763449.535156:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504351872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763449.535183:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763449.535202:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763449/real 1789763449] req@ffff9018d070ed80 x1876699504351872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763449.535246:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504351872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763449.535264:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763449.535279:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763449.535286:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504351872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763449.535296:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504352000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763449.535326:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763449.535342:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763449/real 1789763449] req@ffff9018d070ce00 x1876699504352000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763449.535357:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504352000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763449.535367:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763449.535371:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763449.535375:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504352000:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763450.174297:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763451.198282:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763452.222229:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763453.246288:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763454.270247:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763454.654143:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763454.654154:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763454.654159:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763454.654170:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3381 00000100:00080000:3.0:1789763454.654183:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763454.654191:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763454.654237:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504352128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763454.654271:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763454.654276:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763454.654279:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763454.654285:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3381 00000100:00080000:3.0:1789763454.654289:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763454.654294:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763454.654306:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763454.654319:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504352256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763454.654332:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763454.654335:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763454.654337:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763454.654341:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3381 00000100:00080000:3.0:1789763454.654344:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763454.654349:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763454.654354:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763454.654365:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504352384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763454.654375:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763454.654380:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763454.654382:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763454.654387:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3381 00000100:00080000:3.0:1789763454.654391:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763454.654398:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504352128:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763454.654400:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763454.654416:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763454.654433:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504352512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763454.654448:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763454.654453:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763454.654455:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763454.654460:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3381 00000100:00080000:3.0:1789763454.654465:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763454.654472:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763454.654481:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763454.654491:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763454.654505:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504352640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763454.654524:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763454.654532:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763454.654536:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763454.654545:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763454/real 1789763454] req@ffff9018d070f480 x1876699504352128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763454.654547:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3381 00000100:00080000:3.0:1789763454.654562:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763454.654577:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504352128/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763454.654595:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763454.654601:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789763454.654607:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763454.654611:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763454.654621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504352128:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763454.654654:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504352256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763454.654655:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504352768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763454.654697:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763454.654771:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763454/real 1789763454] req@ffff9018d070fb80 x1876699504352256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763454.654798:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504352256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763454.654826:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763454.654833:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763454.654840:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504352256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763454.654851:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504352384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763454.654891:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763454.654914:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763454/real 1789763454] req@ffff9018d070df80 x1876699504352384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763454.654938:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504352384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763454.654953:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763454.654959:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763454.654965:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504352384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763454.655008:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504352512:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763454.655089:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763454.655107:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763454/real 1789763454] req@ffff9018d070ce00 x1876699504352512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763454.655122:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504352512/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763454.655132:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763454.655135:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763454.655140:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504352512:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763454.655147:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504352640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763454.655167:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763454.655178:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763454/real 1789763454] req@ffff9018d070ed80 x1876699504352640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763454.655191:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504352640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763454.655199:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763454.655202:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763454.655206:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504352640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763454.655212:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504352768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763454.655228:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763454.655237:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763454/real 1789763454] req@ffff9018d070d880 x1876699504352768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763454.655250:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504352768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763454.655259:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763454.655261:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763454.655265:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504352768:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763455.294327:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763456.318234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763457.342293:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763458.366273:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763459.390342:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763459.774137:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763459.774146:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763459.774150:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763459.774157:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3386 00000100:00080000:3.0:1789763459.774167:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763459.774172:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763459.774204:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504352896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763459.774227:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763459.774230:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763459.774231:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763459.774234:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3386 00000100:00080000:3.0:1789763459.774237:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763459.774239:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763459.774247:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763459.774255:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504353024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763459.774265:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763459.774268:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763459.774269:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763459.774271:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3386 00000100:00080000:3.0:1789763459.774273:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763459.774275:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763459.774278:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763459.774284:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504353152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763459.774290:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763459.774293:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763459.774294:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763459.774297:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3386 00000100:00080000:3.0:1789763459.774299:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763459.774302:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763459.774305:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763459.774313:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504352896:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763459.774313:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504353280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763459.774326:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763459.774328:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763459.774329:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763459.774332:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3386 00000100:00080000:3.0:1789763459.774333:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763459.774335:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763459.774338:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763459.774345:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504353408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763459.774351:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763459.774353:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763459.774354:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763459.774356:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3386 00000100:00080000:3.0:1789763459.774358:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763459.774360:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763459.774364:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763459.774370:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504353536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763459.774385:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763459.774424:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763459/real 1789763459] req@ffff9018d070ca80 x1876699504352896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763459.774440:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504352896/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763459.774447:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763459.774450:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763459.774453:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504352896:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763459.774458:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504353024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763459.774473:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763459.774481:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763459/real 1789763459] req@ffff9018d070d180 x1876699504353024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763459.774489:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504353024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763459.774494:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763459.774497:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763459.774499:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504353024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763459.774502:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504353152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763459.774512:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763459.774517:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763459/real 1789763459] req@ffff9018d070e680 x1876699504353152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763459.774524:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504353152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763459.774533:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763459.774535:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763459.774537:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504353152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763459.774556:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504353280:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763459.774567:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763459.774572:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763459/real 1789763459] req@ffff9018d070f100 x1876699504353280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763459.774579:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504353280/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763459.774584:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763459.774586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763459.774588:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504353280:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763459.774592:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504353408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763459.774601:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763459.774607:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763459/real 1789763459] req@ffff9018d070f800 x1876699504353408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763459.774635:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504353408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763459.774645:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763459.774649:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763459.774653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504353408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763459.774660:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504353536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763459.774680:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763459.774691:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763459/real 1789763459] req@ffff9018d070d500 x1876699504353536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763459.774733:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504353536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763459.774758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763459.774760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763459.774762:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504353536:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763460.414207:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763461.438317:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763462.462311:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763463.486247:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763464.510344:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763464.894214:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763464.894228:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763464.894233:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763464.894247:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3391 00000100:00080000:3.0:1789763464.894263:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763464.894274:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763464.894325:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504353664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763464.894358:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763464.894363:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763464.894366:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763464.894371:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3391 00000100:00080000:3.0:1789763464.894375:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763464.894380:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763464.894394:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763464.894411:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504353792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763464.894430:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763464.894434:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789763464.894437:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504353664:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763464.894438:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763464.894454:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3391 00000100:00080000:3.0:1789763464.894459:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763464.894463:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763464.894469:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763464.894486:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504353920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763464.894498:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763464.894502:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763464.894505:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763464.894510:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3391 00000100:00080000:3.0:1789763464.894514:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763464.894520:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763464.894527:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763464.894541:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763464.894548:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504354048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763464.894564:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763464.894569:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763464.894572:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763464.894577:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3391 00000100:00080000:3.0:1789763464.894581:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763464.894586:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763464.894594:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763464.894607:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763464/real 1789763464] req@ffff9018d070dc00 x1876699504353664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763464.894614:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504354176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763464.894684:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763464.894689:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763464.894693:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763464.894701:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504353664/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763464.894705:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3391 00000100:00080000:3.0:1789763464.894717:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763464.894729:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763464.894745:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763464.894750:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763464.894765:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763464.894775:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504354304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763464.894784:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504353664:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763464.894836:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504353792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763464.894883:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763464.894905:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763464/real 1789763464] req@ffff9018d070c000 x1876699504353792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763464.894924:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504353792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763464.894941:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763464.894949:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763464.894954:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504353792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763464.894963:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504353920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763464.894999:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763464.895016:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763464/real 1789763464] req@ffff9018d070e300 x1876699504353920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763464.895089:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504353920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763464.895103:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763464.895108:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763464.895116:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504353920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763464.895127:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504354048:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763464.895175:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763464.895206:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763464/real 1789763464] req@ffff9018d070d500 x1876699504354048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763464.895223:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504354048/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763464.895234:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763464.895242:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763464.895247:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504354048:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763464.895255:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504354176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763464.895279:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763464.895299:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763464/real 1789763464] req@ffff9018d070f800 x1876699504354176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763464.895321:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504354176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763464.895334:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763464.895337:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763464.895341:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504354176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763464.895352:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504354304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763464.895391:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763464.895414:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763464/real 1789763464] req@ffff9018d070f100 x1876699504354304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763464.895443:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504354304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763464.895461:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763464.895465:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763464.895471:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504354304:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763465.534200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763466.558273:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763467.582226:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763468.606249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763469.630261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763470.014167:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763470.014181:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763470.014186:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763470.014204:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3396 00000100:00080000:3.0:1789763470.014219:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763470.014228:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763470.014284:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504354432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763470.014333:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763470.014341:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763470.014345:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763470.014355:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3396 00000100:00080000:3.0:1789763470.014362:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763470.014369:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763470.014383:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763470.014395:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504354432:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763470.014409:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504354560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763470.014427:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763470.014432:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763470.014435:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763470.014441:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3396 00000100:00080000:3.0:1789763470.014446:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763470.014450:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763470.014457:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763470.014472:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504354688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763470.014484:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789763470.014488:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763470.014491:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763470.014497:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763470.014505:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3396 00000100:00080000:3.0:1789763470.014512:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763470.014517:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763470.014525:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763470.014541:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504354816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763470.014555:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763470.014559:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763470.014562:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763470.014568:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763470/real 1789763470] req@ffff9018d070e680 x1876699504354432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763470.014571:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3396 00000100:00080000:3.0:1789763470.014582:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763470.014594:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763470.014607:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763470.014620:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504354432/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763470.014668:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504354944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763470.014706:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763470.014708:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763470.014713:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763470.014716:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763470.014719:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789763470.014725:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3396 00000100:00100000:1.0:1789763470.014731:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504354432:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763470.014732:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763470.014742:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763470.014748:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763470.014760:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504355072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763470.014766:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504354560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763470.014807:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763470.014821:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763470/real 1789763470] req@ffff9018d070d180 x1876699504354560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763470.014834:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504354560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763470.014842:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763470.014845:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763470.014848:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504354560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763470.014854:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504354688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763470.014869:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763470.014878:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763470/real 1789763470] req@ffff9018d070ca80 x1876699504354688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763470.014888:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504354688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763470.014896:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763470.014898:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763470.014901:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504354688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763470.014906:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504354816:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763470.014920:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763470.014929:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763470/real 1789763470] req@ffff9018d070d880 x1876699504354816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763470.014939:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504354816/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763470.014946:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763470.014949:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763470.014951:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504354816:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763470.014956:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504354944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763470.014987:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763470.015000:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763470/real 1789763470] req@ffff9018d070ed80 x1876699504354944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763470.015014:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504354944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763470.015025:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763470.015028:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763470.015032:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504354944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763470.015055:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504355072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763470.015111:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763470.015127:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763470/real 1789763470] req@ffff9018d070ce00 x1876699504355072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763470.015141:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504355072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763470.015152:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763470.015156:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763470.015160:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504355072:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763470.654215:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763471.678277:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763472.702157:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763473.726215:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763474.750395:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763475.134168:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763475.134192:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763475.134197:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763475.134209:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3401 00000100:00080000:2.0:1789763475.134222:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763475.134230:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763475.134295:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504355200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763475.134331:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763475.134336:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763475.134338:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763475.134344:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3401 00000100:00080000:2.0:1789763475.134349:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763475.134354:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763475.134384:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763475.134397:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504355328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763475.134407:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763475.134410:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763475.134413:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763475.134417:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3401 00000100:00080000:2.0:1789763475.134420:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763475.134424:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763475.134429:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763475.134439:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504355456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763475.134449:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763475.134453:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763475.134476:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763475.134478:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504355200:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763475.134486:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3401 00000100:00080000:2.0:1789763475.134496:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763475.134500:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763475.134512:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763475.134530:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504355584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763475.134544:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763475.134565:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763475.134567:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763475.134571:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3401 00000100:00080000:2.0:1789763475.134575:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763475.134579:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763475.134584:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763475.134623:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763475.134626:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504355712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763475.134664:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763475.134670:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763475.134673:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763475.134681:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3401 00000100:00080000:2.0:1789763475.134689:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763475.134698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763475/real 1789763475] req@ffff9018d0f27100 x1876699504355200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763475.134701:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763475.134720:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763475.134735:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504355200/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763475.134749:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504355840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763475.134759:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763475.134771:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763475.134781:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504355200:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763475.134805:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504355328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763475.134850:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763475.134864:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763475/real 1789763475] req@ffff9018d0f25f80 x1876699504355328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763475.134877:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504355328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763475.134885:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763475.134889:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763475.134892:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504355328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763475.134917:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504355456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763475.134935:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763475.134944:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763475/real 1789763475] req@ffff9018d0f26d80 x1876699504355456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763475.134956:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504355456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763475.134963:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763475.134966:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763475.134969:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504355456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763475.134975:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504355584:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763475.134990:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763475.134999:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763475/real 1789763475] req@ffff9018d0f26300 x1876699504355584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763475.135011:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504355584/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763475.135019:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763475.135021:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763475.135024:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504355584:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763475.135029:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504355712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763475.135043:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763475.135090:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763475/real 1789763475] req@ffff9018d0f25180 x1876699504355712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763475.135103:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504355712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763475.135128:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763475.135132:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763475.135135:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504355712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763475.135142:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504355840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763475.135167:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763475.135183:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763475/real 1789763475] req@ffff9018d0f24380 x1876699504355840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763475.135196:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504355840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763475.135205:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763475.135209:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763475.135212:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504355840:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763475.774247:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763476.798263:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763477.822334:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763478.846272:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763479.870282:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763480.254274:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763480.254287:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763480.254292:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763480.254304:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3406 00000100:00080000:3.0:1789763480.254318:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763480.254328:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763480.254375:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504355968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763480.254437:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763480.254455:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763480.254460:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763480.254469:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3406 00000100:00080000:3.0:1789763480.254476:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763480.254486:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763480.254504:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763480.254535:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504355968:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763480.254535:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504356096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763480.254600:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763480.254606:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763480.254608:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763480.254614:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3406 00000100:00080000:3.0:1789763480.254619:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763480.254626:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763480.254634:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763480.254653:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504356224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763480.254676:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789763480.254678:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763480.254686:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763480.254690:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763480.254698:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3406 00000100:00080000:3.0:1789763480.254705:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763480.254710:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763480.254717:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763480.254744:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504356352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763480.254758:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763480/real 1789763480] req@ffff9018d070df80 x1876699504355968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763480.254762:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763480.254776:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763480.254780:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763480.254786:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3406 00000100:00080000:3.0:1789763480.254790:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763480.254795:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763480.254808:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763480.254832:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504355968/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763480.254834:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504356480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763480.254854:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763480.254873:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789763480.254882:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763480.254885:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504355968:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763480.254892:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763480.254903:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763480.254914:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3406 00000100:00080000:3.0:1789763480.254920:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763480.254926:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763480.254931:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763480.254943:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504356096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763480.254950:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504356608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763480.254996:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763480.255017:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763480/real 1789763480] req@ffff9018d070fb80 x1876699504356096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763480.255072:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504356096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763480.255084:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763480.255089:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763480.255093:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504356096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763480.255102:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504356224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763480.255128:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763480.255147:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763480/real 1789763480] req@ffff9018d070f480 x1876699504356224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763480.255162:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504356224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763480.255171:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763480.255175:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763480.255179:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504356224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763480.255186:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504356352:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763480.255205:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763480.255220:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763480/real 1789763480] req@ffff9018d070ce00 x1876699504356352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763480.255244:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504356352/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763480.255260:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763480.255267:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763480.255274:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504356352:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763480.255285:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504356480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763480.255327:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763480.255345:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763480/real 1789763480] req@ffff9018d070ed80 x1876699504356480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763480.255360:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504356480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763480.255371:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763480.255374:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763480.255378:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504356480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763480.255403:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504356608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763480.255426:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763480.255446:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763480/real 1789763480] req@ffff9018d070d880 x1876699504356608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763480.255461:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504356608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763480.255470:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763480.255473:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763480.255477:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504356608:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763480.894187:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763481.918208:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763482.942331:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763483.966297:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763484.990217:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763485.374145:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763485.374158:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763485.374164:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763485.374178:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3411 00000100:00080000:3.0:1789763485.374209:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763485.374220:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763485.374273:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504356736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763485.374333:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763485.374338:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763485.374341:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763485.374346:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3411 00000100:00080000:3.0:1789763485.374359:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763485.374365:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763485.374379:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763485.374399:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504356864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763485.374423:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763485.374428:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763485.374430:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763485.374437:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3411 00000100:00080000:3.0:1789763485.374443:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763485.374448:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504356736:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763485.374451:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763485.374463:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763485.374476:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504356992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763485.374508:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763485.374536:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763485.374539:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763485.374545:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3411 00000100:00080000:3.0:1789763485.374551:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763485.374556:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763485.374562:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763485.374578:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504357120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763485.374592:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763485.374596:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763485.374599:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763485.374603:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3411 00000100:00080000:3.0:1789763485.374608:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763485.374615:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789763485.374626:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789763485.374626:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763485.374646:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504357248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763485.374658:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763485.374663:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763485.374665:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763485.374669:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3411 00000100:00080000:3.0:1789763485.374673:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763485.374678:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763485.374684:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763485.374706:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504357376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763485.374708:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763485/real 1789763485] req@ffff9018d070ca80 x1876699504356736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763485.374761:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504356736/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763485.374792:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763485.374801:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763485.374809:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504356736:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763485.374840:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504356864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763485.374889:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763485.374918:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763485/real 1789763485] req@ffff9018d070d180 x1876699504356864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763485.374962:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504356864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763485.374978:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763485.374984:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763485.374990:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504356864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763485.375029:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504356992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763485.375139:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763485.375170:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763485/real 1789763485] req@ffff9018d070e680 x1876699504356992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763485.375197:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504356992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763485.375231:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763485.375237:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763485.375244:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504356992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763485.375253:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504357120:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763485.375283:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763485.375317:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763485/real 1789763485] req@ffff9018d070f100 x1876699504357120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763485.375332:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504357120/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763485.375343:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763485.375347:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763485.375350:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504357120:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763485.375357:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504357248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763485.375378:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763485.375389:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763485/real 1789763485] req@ffff9018d070f800 x1876699504357248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763485.375405:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504357248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763485.375421:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763485.375427:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763485.375433:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504357248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763485.375445:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504357376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763485.375500:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763485.375547:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763485/real 1789763485] req@ffff9018d070d500 x1876699504357376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763485.375571:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504357376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763485.375585:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763485.375591:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763485.375597:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504357376:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763486.014333:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763487.038220:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763488.062224:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763489.086224:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763490.110254:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763490.494135:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763490.494148:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763490.494154:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763490.494183:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3417 00000100:00080000:3.0:1789763490.494194:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763490.494201:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763490.494240:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504357504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763490.494264:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763490.494267:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763490.494269:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763490.494272:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3417 00000100:00080000:3.0:1789763490.494275:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763490.494278:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763490.494307:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763490.494326:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504357632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763490.494342:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504357504:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763490.494346:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763490.494357:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763490.494360:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763490.494366:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3417 00000100:00080000:3.0:1789763490.494371:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763490.494376:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763490.494383:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763490.494401:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504357760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763490.494413:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763490.494418:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763490.494421:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763490.494426:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3417 00000100:00080000:3.0:1789763490.494446:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763490.494450:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763490.494457:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763490.494493:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504357888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763490.494511:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763490.494519:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789763490.494536:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504357632:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789763490.494540:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763490.494547:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3417 00000100:00080000:3.0:1789763490.494551:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763490.494556:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763490.494562:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763490.494570:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504357760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763490.494576:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504358016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763490.494587:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504357888:192.168.203.158@tcp:8:kworker.0 00000100:00080000:3.0:1789763490.494588:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763490.494595:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763490.494614:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763490.494622:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3417 00000100:00080000:3.0:1789763490.494630:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763490.494636:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763490.494640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504358016:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789763490.494647:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763490.494671:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504358144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763490.494702:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504358144:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763491.134192:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763492.158306:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763493.182242:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763493.566482:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789763493.566550:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763493.566580:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763493.566633:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763493.566660:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789763493.566672:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763490/real 1789763493] req@ffff9018d070e300 x1876699504357504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1789763493.566683:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763493.566702:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763493.566728:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789763493.566735:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504357504/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763493.566758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763493.566764:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763493.566771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504357504:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763493.566810:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763490/real 1789763493] req@ffff9018d070f100 x1876699504358144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763493.566845:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504358144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763493.566864:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763493.566872:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763493.566879:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504358144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763493.566892:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763490/real 1789763493] req@ffff9018d070f800 x1876699504358016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763493.566915:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504358016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763493.566931:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763493.566938:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763493.566945:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504358016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763493.566957:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763490/real 1789763493] req@ffff9018d070c000 x1876699504357632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763493.566982:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504357632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763493.566998:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763493.567003:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763493.567008:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504357632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763493.567017:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763490/real 1789763493] req@ffff9018d070dc00 x1876699504357760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763493.567087:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504357760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763493.567109:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763493.567130:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763493.567140:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504357760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763493.567154:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763490/real 1789763493] req@ffff9018d070d500 x1876699504357888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763493.567182:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504357888/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763493.567198:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763493.567204:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763493.567211:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504357888:192.168.203.158@tcp:8:kworker.0 00010000:00010000:3.0:1789763494.206203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763495.230256:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763495.614207:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763495.614220:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763495.614226:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763495.614239:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3422 00000100:00080000:2.0:1789763495.614254:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763495.614263:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763495.614308:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504358272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763495.614345:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763495.614350:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763495.614352:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763495.614357:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3422 00000100:00080000:2.0:1789763495.614361:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763495.614366:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763495.614378:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763495.614399:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504358400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763495.614415:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763495.614423:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504358272:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763495.614424:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763495.614433:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763495.614472:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3422 00000100:00080000:2.0:1789763495.614477:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763495.614483:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763495.614492:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763495.614513:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504358528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763495.614527:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763495.614532:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763495.614535:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763495.614539:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3422 00000100:00080000:2.0:1789763495.614544:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763495.614549:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789763495.614561:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789763495.614563:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763495.614581:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504358656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763495.614596:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763495.614601:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763495.614603:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763495.614609:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3422 00000100:00080000:2.0:1789763495.614615:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763495.614620:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763495.614627:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763495.614637:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763495/real 1789763495] req@ffff9018d0f24a80 x1876699504358272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763495.614659:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504358784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763495.614683:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504358272/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763495.614686:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763495.614703:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763495.614711:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763495.614723:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763495.614725:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3422 00000100:00080000:1.0:1789763495.614739:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763495.614740:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763495.614752:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504358272:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763495.614761:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763495.614778:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763495.614802:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504358912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763495.614820:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504358400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763495.614889:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763495.614917:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763495/real 1789763495] req@ffff9018d0f26a00 x1876699504358400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763495.614944:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504358400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763495.614956:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763495.614962:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763495.614970:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504358400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763495.614984:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504358528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763495.615027:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763495.615108:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763495/real 1789763495] req@ffff9018d0f25c00 x1876699504358528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763495.615135:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504358528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763495.615151:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763495.615157:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763495.615164:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504358528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763495.615177:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504358656:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763495.615222:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763495.615247:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763495/real 1789763495] req@ffff9018d0f27800 x1876699504358656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763495.615272:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504358656/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763495.615288:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763495.615295:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763495.615302:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504358656:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763495.615314:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504358784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763495.615354:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763495.615376:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763495/real 1789763495] req@ffff9018d0f24000 x1876699504358784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763495.615399:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504358784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763495.615414:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763495.615420:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763495.615427:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504358784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763495.615497:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504358912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763495.615539:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763495.615563:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763495/real 1789763495] req@ffff9018d0f26680 x1876699504358912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763495.615604:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504358912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763495.615623:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763495.615630:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763495.615636:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504358912:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763496.254345:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763497.278150:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763498.302250:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763499.326199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763500.350299:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763500.734267:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763500.734278:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763500.734284:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763500.734297:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3427 00000100:00080000:0.0:1789763500.734311:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763500.734320:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763500.734370:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504359040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763500.734431:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763500.734437:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763500.734440:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763500.734449:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3427 00000100:00080000:0.0:1789763500.734455:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763500.734460:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763500.734476:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763500.734516:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504359168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763500.734534:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763500.734542:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789763500.734547:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504359040:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763500.734548:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763500.734562:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3427 00000100:00080000:0.0:1789763500.734571:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763500.734577:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763500.734585:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763500.734605:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504359296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763500.734620:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763500.734625:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763500.734628:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763500.734633:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3427 00000100:00080000:0.0:1789763500.734638:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763500.734644:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763500.734652:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789763500.734664:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763500.734673:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504359424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763500.734689:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763500.734694:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763500.734697:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763500.734702:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3427 00000100:00080000:0.0:1789763500.734707:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763500.734713:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763500.734719:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763500.734730:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763500/real 1789763500] req@ffff9018c8b99880 x1876699504359040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763500.734738:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504359552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763500.734786:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763500.734798:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504359040/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763500.734799:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763500.734808:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763500.734826:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3427 00000100:00080000:2.0:1789763500.734839:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763500.734840:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763500.734854:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789763500.734860:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:0.0:1789763500.734870:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763500.734875:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504359040:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763500.734903:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504359680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763500.734922:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504359168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763500.734967:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763500.734986:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763500/real 1789763500] req@ffff9018c8b98000 x1876699504359168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763500.735004:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504359168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763500.735016:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763500.735020:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763500.735025:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504359168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763500.735031:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504359296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763500.735112:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763500.735132:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763500/real 1789763500] req@ffff9018c8b9b800 x1876699504359296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763500.735149:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504359296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763500.735160:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763500.735164:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763500.735169:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504359296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763500.735179:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504359424:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763500.735207:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763500.735225:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763500/real 1789763500] req@ffff9018c8b9aa00 x1876699504359424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763500.735244:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504359424/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763500.735256:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763500.735262:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763500.735268:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504359424:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763500.735281:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504359552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763500.735310:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763500.735330:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763500/real 1789763500] req@ffff9018c8b98e00 x1876699504359552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763500.735350:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504359552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763500.735361:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763500.735366:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763500.735370:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504359552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763500.735392:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504359680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763500.735443:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763500.735458:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763500/real 1789763500] req@ffff9018c8b9a680 x1876699504359680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763500.735482:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504359680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763500.735492:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763500.735496:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763500.735500:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504359680:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763501.374207:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763502.398251:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763503.422255:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763504.446262:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763505.470267:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763505.854219:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763505.854231:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763505.854236:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763505.854247:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3432 00000100:00080000:1.0:1789763505.854259:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763505.854267:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763505.854314:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504359808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763505.854349:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763505.854356:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763505.854376:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763505.854386:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3432 00000100:00080000:1.0:1789763505.854392:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763505.854416:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763505.854433:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763505.854448:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504359808:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763505.854457:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504359936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763505.854488:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763505.854494:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763505.854497:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763505.854504:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3432 00000100:00080000:1.0:1789763505.854509:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763505.854533:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763505.854539:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763505.854551:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763505.854554:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504360064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763505.854568:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763505.854573:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763505.854576:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763505.854581:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3432 00000100:00080000:1.0:1789763505.854586:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763505.854592:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763505.854599:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763505.854602:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763505/real 1789763505] req@ffff9018c66a5c00 x1876699504359808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763505.854614:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504360192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763505.854628:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504359808/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763505.854629:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763505.854635:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763505.854639:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763505.854644:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3432 00000100:00080000:2.0:1789763505.854645:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763505.854650:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763505.854651:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789763505.854656:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763505.854658:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504359808:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789763505.854663:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763505.854676:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504360320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763505.854691:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763505.854692:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504359936:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789763505.854697:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763505.854700:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763505.854706:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3432 00000100:00080000:1.0:1789763505.854711:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763505.854716:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763505.854723:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763505.854739:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504360448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763505.854744:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763505.854766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763505/real 1789763505] req@ffff9018c66a6d80 x1876699504359936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763505.854786:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504359936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763505.854801:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763505.854806:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763505.854813:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504359936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763505.854824:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504360064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763505.854855:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763505.854870:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763505/real 1789763505] req@ffff9018c66a5f80 x1876699504360064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763505.854882:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504360064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763505.854891:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763505.854895:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763505.854898:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504360064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763505.854904:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504360192:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763505.854921:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763505.854931:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763505/real 1789763505] req@ffff9018c66a4000 x1876699504360192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763505.854943:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504360192/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763505.854951:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763505.854954:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763505.854958:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504360192:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763505.854978:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504360320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763505.854995:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763505.855005:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763505/real 1789763505] req@ffff9018c66a7b80 x1876699504360320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763505.855016:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504360320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763505.855024:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763505.855027:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763505.855030:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504360320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763505.855067:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504360448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763505.855087:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763505.855099:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763505/real 1789763505] req@ffff9018c66a7800 x1876699504360448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763505.855111:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504360448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763505.855120:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763505.855123:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763505.855126:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504360448:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763506.494332:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763507.518386:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763508.542339:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763509.566281:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763510.590310:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763510.974153:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763510.974169:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763510.974176:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763510.974192:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3437 00000100:00080000:1.0:1789763510.974210:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763510.974222:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763510.974287:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504360576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763510.974354:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763510.974364:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763510.974369:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763510.974377:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3437 00000100:00080000:1.0:1789763510.974384:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763510.974391:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763510.974407:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763510.974412:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504360576:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763510.974433:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504360704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763510.974450:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763510.974455:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763510.974458:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763510.974463:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3437 00000100:00080000:1.0:1789763510.974469:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763510.974473:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763510.974479:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763510.974497:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504360832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763510.974510:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763510.974515:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763510.974517:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763510.974527:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3437 00000800:00000100:2.0:1789763510.974528:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763510.974534:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763510.974541:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763510.974550:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763510.974567:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504360960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763510.974580:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763510.974584:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763510.974587:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763510.974593:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3437 00000100:00080000:1.0:1789763510.974598:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763510.974604:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763510/real 1789763510] req@ffff9018c66a5180 x1876699504360576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763510.974610:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763510.974622:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763510.974648:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504361088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763510.974654:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504360576/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763510.974684:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763510.974692:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763510.974695:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763510.974696:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763510.974704:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3437 00000100:00080000:2.0:1789763510.974707:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789763510.974713:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763510.974720:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504360576:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763510.974727:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763510.974738:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763510.974752:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504361216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763510.974785:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504360704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763510.974819:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763510.974837:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763510/real 1789763510] req@ffff9018c66a6300 x1876699504360704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763510.974854:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504360704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763510.974866:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763510.974871:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763510.974875:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504360704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763510.974886:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504360832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763510.974923:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763510.974944:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763510/real 1789763510] req@ffff9018c66a4700 x1876699504360832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763510.974984:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504360832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763510.974999:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763510.975003:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763510.975008:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504360832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763510.975016:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504360960:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763510.975100:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763510.975117:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763510/real 1789763510] req@ffff9018c66a6680 x1876699504360960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763510.975133:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504360960/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763510.975143:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763510.975147:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763510.975152:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504360960:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763510.975159:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504361088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763510.975181:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763510.975192:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763510/real 1789763510] req@ffff9018c66a5500 x1876699504361088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763510.975206:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504361088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763510.975224:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763510.975228:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763510.975233:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504361088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763510.975239:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504361216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763510.975258:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763510.975268:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763510/real 1789763510] req@ffff9018c66a4e00 x1876699504361216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763510.975281:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504361216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763510.975291:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763510.975295:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763510.975298:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504361216:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763511.614361:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763512.638197:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763513.662336:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763514.686225:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763515.710239:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763516.094228:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763516.094245:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763516.094252:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763516.094268:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3442 00000100:00080000:2.0:1789763516.094306:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763516.094318:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763516.094383:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504361344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763516.094436:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763516.094443:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763516.094446:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763516.094452:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3442 00000100:00080000:2.0:1789763516.094459:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763516.094465:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763516.094482:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763516.094500:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504361472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763516.094515:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504361344:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763516.094522:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763516.094537:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763516.094540:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763516.094547:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3442 00000100:00080000:2.0:1789763516.094552:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763516.094558:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763516.094565:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763516.094584:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504361600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763516.094597:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763516.094602:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763516.094605:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763516.094610:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3442 00000100:00080000:2.0:1789763516.094628:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763516.094634:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763516.094643:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763516.094646:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763516.094663:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504361728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763516.094682:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763516.094687:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763516.094691:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763516.094695:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3442 00000100:00080000:2.0:1789763516.094699:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763516.094705:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763516.094711:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763516.094731:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763516/real 1789763516] req@ffff9018d0f24e00 x1876699504361344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763516.094745:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504361856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763516.094793:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763516.094806:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504361344/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763516.094808:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763516.094817:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763516.094843:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3442 00000100:00080000:1.0:1789763516.094868:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763516.094876:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763516.094879:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763516.094895:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763516.094897:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504361344:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789763516.094916:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763516.094945:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504361984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763516.094968:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504361472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763516.095030:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763516.095089:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763516/real 1789763516] req@ffff9018d0f27480 x1876699504361472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763516.095109:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504361472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763516.095121:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763516.095125:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763516.095130:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504361472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763516.095139:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504361600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763516.095167:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763516.095180:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763516/real 1789763516] req@ffff9018d0f25500 x1876699504361600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763516.095194:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504361600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763516.095213:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763516.095218:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763516.095222:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504361600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763516.095229:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504361728:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763516.095250:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763516.095261:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763516/real 1789763516] req@ffff9018d0f24700 x1876699504361728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763516.095275:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504361728/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763516.095313:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763516.095318:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763516.095323:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504361728:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763516.095332:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504361856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763516.095363:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763516.095378:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763516/real 1789763516] req@ffff9018d0f26680 x1876699504361856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763516.095392:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504361856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763516.095402:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763516.095406:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763516.095411:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504361856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763516.095436:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504361984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763516.095457:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763516.095469:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763516/real 1789763516] req@ffff9018d0f24000 x1876699504361984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763516.095483:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504361984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763516.095493:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763516.095496:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763516.095500:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504361984:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763516.734334:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763517.758413:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763518.782250:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763519.806303:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763520.830229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763521.214207:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763521.214232:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763521.214238:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763521.214269:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3447 00000100:00080000:1.0:1789763521.214284:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763521.214304:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763521.214361:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504362112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763521.214413:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763521.214418:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763521.214421:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763521.214429:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3447 00000100:00080000:1.0:1789763521.214438:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763521.214448:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763521.214467:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763521.214490:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504362112:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763521.214502:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504362240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763521.214530:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763521.214537:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763521.214540:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763521.214547:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3447 00000100:00080000:1.0:1789763521.214555:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763521.214561:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763521.214573:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763521.214599:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504362368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763521.214612:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763521.214635:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763521.214643:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763521.214647:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763521.214657:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3447 00000100:00080000:1.0:1789763521.214665:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763521.214675:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763521.214685:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763521.214697:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763521/real 1789763521] req@ffff9018c66a5880 x1876699504362112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763521.214721:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504362496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763521.214729:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504362112/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763521.214813:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763521.214822:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763521.214827:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763521.214827:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763521.214846:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789763521.214848:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3447 00000100:00100000:2.0:1789763521.214861:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504362112:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763521.214867:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763521.214883:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763521.214896:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763521.214914:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504362240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763521.214929:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504362624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763521.214949:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763521.214963:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763521.214969:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763521.214972:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763521.214980:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763521/real 1789763521] req@ffff9018c66a7480 x1876699504362240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763521.214987:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3447 00000100:00080000:1.0:1789763521.215006:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763521.215015:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763521.215025:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504362240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789763521.215027:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763521.215088:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763521.215094:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763521.215099:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504362240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763521.215107:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504362752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763521.215119:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504362368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763521.215183:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763521.215200:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763521/real 1789763521] req@ffff9018c66a4a80 x1876699504362368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763521.215213:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504362368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763521.215220:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763521.215223:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763521.215226:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504362368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763521.215232:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504362496:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763521.215268:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763521.215285:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763521/real 1789763521] req@ffff9018c66a7100 x1876699504362496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763521.215303:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504362496/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763521.215316:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763521.215321:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763521.215326:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504362496:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763521.215349:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504362624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763521.215372:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763521.215382:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763521/real 1789763521] req@ffff9018c66a4380 x1876699504362624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763521.215393:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504362624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763521.215406:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763521.215409:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763521.215412:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504362624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763521.215417:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504362752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763521.215432:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763521.215440:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763521/real 1789763521] req@ffff9018c66a6a00 x1876699504362752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763521.215450:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504362752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763521.215457:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763521.215459:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763521.215462:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504362752:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763521.854332:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763522.878335:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763523.902354:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763524.926357:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763525.950339:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763526.334270:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763526.334289:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763526.334297:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763526.334316:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3452 00000100:00080000:2.0:1789763526.334335:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763526.334366:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763526.334463:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504362880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763526.334502:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763526.334507:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763526.334509:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763526.334515:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3452 00000100:00080000:2.0:1789763526.334521:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763526.334533:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763526.334547:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763526.334564:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504362880:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763526.334569:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504363008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763526.334591:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763526.334595:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763526.334598:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763526.334602:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3452 00000100:00080000:2.0:1789763526.334606:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763526.334610:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763526.334616:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763526.334628:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504363136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763526.334639:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763526.334643:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763526.334645:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763526.334649:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3452 00000100:00080000:2.0:1789763526.334655:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763526.334659:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763526.334665:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763526.334677:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504363264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763526.334688:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763526.334693:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789763526.334700:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763526.334701:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763526.334712:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3452 00000100:00080000:2.0:1789763526.334718:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763526.334723:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763526.334729:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763526.334742:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504363392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763526.334752:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763526.334758:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763526.334760:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763526.334765:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3452 00000100:00080000:2.0:1789763526.334777:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763526.334779:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763526/real 1789763526] req@ffff9018d0f27800 x1876699504362880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763526.334785:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763526.334796:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763526.334818:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504363520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763526.334824:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504362880/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763526.334897:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763526.334903:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763526.334910:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504362880:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763526.334946:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504363008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763526.334974:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763526.335007:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763526/real 1789763526] req@ffff9018d0f25c00 x1876699504363008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763526.335023:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504363008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763526.335034:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763526.335038:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763526.335043:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504363008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763526.335104:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504363136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763526.335147:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763526.335172:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763526/real 1789763526] req@ffff9018d0f26a00 x1876699504363136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763526.335233:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504363136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763526.335248:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763526.335254:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763526.335260:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504363136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763526.335272:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504363264:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763526.335310:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763526.335348:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763526/real 1789763526] req@ffff9018d0f24a80 x1876699504363264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763526.335374:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504363264/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763526.335392:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763526.335396:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763526.335401:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504363264:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763526.335409:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504363392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763526.335459:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763526.335471:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763526/real 1789763526] req@ffff9018d0f24380 x1876699504363392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763526.335484:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504363392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763526.335493:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763526.335497:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763526.335500:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504363392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763526.335506:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504363520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763526.335535:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763526.335553:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763526/real 1789763526] req@ffff9018d0f25180 x1876699504363520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763526.335585:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504363520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763526.335600:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763526.335605:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763526.335610:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504363520:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763526.974299:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763527.998342:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763529.022174:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763530.046275:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763531.070265:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763531.454212:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763531.454229:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763531.454236:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763531.454253:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3458 00000100:00080000:3.0:1789763531.454271:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763531.454283:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763531.454331:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504363648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763531.454371:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763531.454379:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763531.454383:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763531.454392:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3458 00000100:00080000:3.0:1789763531.454400:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763531.454408:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763531.454425:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763531.454448:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504363648:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763531.454449:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504363776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763531.454470:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763531.454476:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763531.454478:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763531.454485:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3458 00000100:00080000:3.0:1789763531.454491:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763531.454498:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763531.454506:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763531.454524:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504363904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763531.454540:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763531.454547:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763531.454549:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763531.454556:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3458 00000800:00000100:1.0:1789763531.454558:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763531.454561:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763531.454581:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763531.454591:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763531.454609:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504364032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763531.454618:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763531/real 1789763531] req@ffff9018d070e680 x1876699504363648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763531.454628:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763531.454636:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763531.454640:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763531.454647:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3458 00000100:00100000:1.0:1789763531.454650:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504363648/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763531.454653:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763531.454660:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763531.454669:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763531.454670:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763531.454677:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763531.454685:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504363648:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763531.454687:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504364160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763531.454706:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763531.454712:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763531.454715:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763531.454721:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3458 00000100:00080000:3.0:1789763531.454726:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763531.454733:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763531.454734:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504363776:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789763531.454742:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763531.454758:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504364288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763531.454775:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763531.454799:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763531/real 1789763531] req@ffff9018d070d180 x1876699504363776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763531.454825:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504363776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763531.454857:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763531.454863:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763531.454871:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504363776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763531.454880:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504363904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763531.454924:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763531.454940:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763531/real 1789763531] req@ffff9018d070ca80 x1876699504363904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763531.454955:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504363904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763531.454966:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763531.454969:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763531.454973:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504363904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763531.454980:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504364032:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763531.455001:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763531.455012:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763531/real 1789763531] req@ffff9018d070d880 x1876699504364032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763531.455026:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504364032/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763531.455072:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763531.455077:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763531.455082:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504364032:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763531.455090:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504364160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763531.455114:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763531.455128:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763531/real 1789763531] req@ffff9018d070ed80 x1876699504364160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763531.455143:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504364160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763531.455152:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763531.455156:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763531.455160:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504364160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763531.455211:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504364288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763531.455236:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763531.455249:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763531/real 1789763531] req@ffff9018d070ce00 x1876699504364288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763531.455263:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504364288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763531.455273:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763531.455276:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763531.455280:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504364288:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763532.094245:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763533.118212:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763534.142225:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763535.166260:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763536.190260:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763536.574089:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763536.574100:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763536.574105:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763536.574117:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3463 00000100:00080000:3.0:1789763536.574128:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763536.574151:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763536.574198:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504364416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763536.574242:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763536.574247:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763536.574249:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763536.574255:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3463 00000100:00080000:3.0:1789763536.574260:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763536.574265:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763536.574278:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763536.574290:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504364416:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763536.574305:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504364544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763536.574326:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763536.574332:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763536.574335:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763536.574344:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3463 00000100:00080000:3.0:1789763536.574361:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763536.574367:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763536.574371:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763536.574380:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763536.574395:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504364672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763536.574406:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763536.574412:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763536/real 1789763536] req@ffff9018d070f480 x1876699504364416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763536.574413:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763536.574418:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763536.574428:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3463 00000100:00080000:3.0:1789763536.574436:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763536.574442:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763536.574448:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763536.574450:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504364416/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763536.574465:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763536.574471:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763536.574477:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504364416:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763536.574505:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504364544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763536.574530:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504364800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763536.574535:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763536.574551:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763536.574555:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763536/real 1789763536] req@ffff9018d070fb80 x1876699504364544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763536.574557:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763536.574561:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763536.574572:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3463 00000100:00080000:3.0:1789763536.574581:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763536.574588:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504364544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763536.574589:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763536.574599:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763536.574604:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763536.574611:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763536.574617:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504364928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763536.574619:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504364544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763536.574633:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504364672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763536.574651:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763536.574660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763536/real 1789763536] req@ffff9018d070df80 x1876699504364672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763536.574671:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504364672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763536.574679:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763536.574682:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763536.574685:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504364672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763536.574718:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504364800:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763536.574740:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763536.574751:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763536/real 1789763536] req@ffff9018d070ce00 x1876699504364800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763536.574765:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504364800/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763536.574775:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763536.574780:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763536.574784:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504364800:192.168.203.158@tcp:8:kworker.0 00000100:00080000:3.0:1789763536.574828:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763536.574833:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763536.574836:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763536.574842:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3463 00000100:00080000:3.0:1789763536.574849:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763536.574855:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763536.574876:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504364928:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789763536.574904:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763536.574925:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504365056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763536.574935:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763536.574967:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763536/real 1789763536] req@ffff9018d070ed80 x1876699504364928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763536.574985:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504364928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763536.574995:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763536.575000:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763536.575005:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504364928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763536.575025:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504365056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763536.575093:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763536.575111:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763536/real 1789763536] req@ffff9018d070d880 x1876699504365056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763536.575124:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504365056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763536.575132:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763536.575150:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763536.575154:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504365056:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763537.214291:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763538.238309:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763539.262199:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763540.286189:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763541.310317:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763541.694183:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763541.694203:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763541.694210:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763541.694226:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3468 00000100:00080000:3.0:1789763541.694244:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763541.694255:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763541.694316:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504365184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763541.694357:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763541.694363:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763541.694365:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763541.694371:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3468 00000100:00080000:3.0:1789763541.694375:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763541.694379:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763541.694393:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763541.694410:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504365312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763541.694425:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763541.694429:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763541.694431:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763541.694461:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3468 00000100:00100000:0.0:1789763541.694466:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504365184:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763541.694468:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763541.694480:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763541.694493:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763541.694509:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504365440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763541.694521:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763541.694526:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763541.694528:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763541.694533:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3468 00000100:00080000:3.0:1789763541.694537:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763541.694541:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763541.694548:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763541.694562:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504365568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763541.694574:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763541.694579:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763541.694582:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763541.694605:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3468 00000100:00080000:3.0:1789763541.694629:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763541.694639:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763541.694644:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763541.694654:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763541.694671:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504365696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763541.694691:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763541.694696:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763541.694698:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763541.694704:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3468 00000100:00080000:3.0:1789763541.694709:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763541.694713:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763541.694719:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763541.694729:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763541/real 1789763541] req@ffff9018d070ca80 x1876699504365184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763541.694738:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504365824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763541.694810:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504365184/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763541.694834:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763541.694842:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763541.694850:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504365184:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763541.694915:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504365312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763541.694962:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763541.695004:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763541/real 1789763541] req@ffff9018d070d180 x1876699504365312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763541.695029:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504365312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763541.695045:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763541.695149:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763541.695157:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504365312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763541.695170:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504365440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763541.695202:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763541.695219:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763541/real 1789763541] req@ffff9018d070e680 x1876699504365440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763541.695242:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504365440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763541.695253:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763541.695256:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763541.695260:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504365440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763541.695266:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504365568:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763541.695286:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763541.695296:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763541/real 1789763541] req@ffff9018d070d500 x1876699504365568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763541.695309:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504365568/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763541.695318:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763541.695322:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763541.695325:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504365568:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763541.695330:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504365696:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763541.695348:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763541.695359:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763541/real 1789763541] req@ffff9018d070dc00 x1876699504365696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763541.695372:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504365696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763541.695389:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763541.695393:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763541.695397:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504365696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763541.695402:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504365824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763541.695421:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763541.695431:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763541/real 1789763541] req@ffff9018d070c000 x1876699504365824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763541.695443:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504365824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763541.695452:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763541.695456:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763541.695459:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504365824:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763542.334263:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763543.358257:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763544.382266:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763545.406293:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763546.430227:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763546.814196:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763546.814208:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763546.814213:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763546.814226:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3473 00000100:00080000:3.0:1789763546.814240:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763546.814249:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763546.814297:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504365952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763546.814334:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763546.814339:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763546.814342:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763546.814347:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3473 00000100:00080000:3.0:1789763546.814353:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763546.814357:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763546.814370:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763546.814385:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504366080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763546.814397:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763546.814401:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763546.814403:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763546.814408:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3473 00000100:00080000:3.0:1789763546.814412:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763546.814417:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763546.814422:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763546.814434:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504366208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763546.814446:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763546.814450:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763546.814453:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763546.814459:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3473 00000100:00080000:3.0:1789763546.814467:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763546.814469:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504365952:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763546.814476:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763546.814491:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763546.814508:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504366336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763546.814530:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763546.814535:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763546.814537:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763546.814542:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3473 00000100:00080000:3.0:1789763546.814546:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763546.814551:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763546.814557:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763546.814576:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504366464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763546.814577:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763546.814595:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763546.814600:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763546.814604:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763546.814610:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3473 00000100:00080000:3.0:1789763546.814614:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763546.814618:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763546.814624:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763546.814640:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504366592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763546.814649:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763546/real 1789763546] req@ffff9018d070f800 x1876699504365952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763546.814691:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504365952/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763546.814713:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763546.814723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763546.814731:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504365952:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763546.814746:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504366080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763546.814791:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763546.814815:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763546/real 1789763546] req@ffff9018d070f100 x1876699504366080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763546.814832:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504366080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763546.814843:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763546.814847:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763546.814852:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504366080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763546.814860:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504366208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763546.814885:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763546.814896:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763546/real 1789763546] req@ffff9018d070e300 x1876699504366208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763546.814911:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504366208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763546.814920:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763546.814924:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763546.814927:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504366208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763546.814959:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504366336:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763546.814981:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763546.814993:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763546/real 1789763546] req@ffff9018d070c000 x1876699504366336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763546.815015:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504366336/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763546.815031:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763546.815112:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763546.815121:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504366336:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763546.815134:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504366464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763546.815168:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763546.815185:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763546/real 1789763546] req@ffff9018d070dc00 x1876699504366464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763546.815200:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504366464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763546.815211:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763546.815215:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763546.815219:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504366464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763546.815225:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504366592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763546.815244:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763546.815254:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763546/real 1789763546] req@ffff9018d070d500 x1876699504366592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763546.815277:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504366592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763546.815286:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763546.815290:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763546.815293:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504366592:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763547.454329:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763548.478240:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763549.502196:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763550.526271:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763551.550275:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763551.934193:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763551.934206:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763551.934212:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763551.934224:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3478 00000100:00080000:1.0:1789763551.934237:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763551.934247:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763551.934297:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504366720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763551.934335:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763551.934340:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763551.934343:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763551.934348:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3478 00000100:00080000:1.0:1789763551.934352:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763551.934356:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763551.934370:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763551.934386:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504366848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763551.934402:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763551.934410:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789763551.934412:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504366720:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763551.934413:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763551.934428:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3478 00000100:00080000:1.0:1789763551.934436:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763551.934450:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763551.934456:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763551.934471:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504366976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763551.934494:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763551.934498:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763551.934502:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763551.934506:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3478 00000100:00080000:1.0:1789763551.934513:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763551.934518:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763551.934523:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789763551.934533:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763551.934541:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504367104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763551.934558:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763551.934563:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763551.934565:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763551.934570:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3478 00000100:00080000:1.0:1789763551.934577:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763551.934583:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763551.934589:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763551.934605:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504367232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763551.934612:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763551/real 1789763551] req@ffff9018c66a6a00 x1876699504366720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763551.934630:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763551.934645:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763551.934648:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763551.934655:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3478 00000100:00080000:1.0:1789763551.934662:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763551.934664:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504366720/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763551.934672:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763551.934686:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789763551.934695:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789763551.934704:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504367360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763551.934708:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763551.934723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504366720:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763551.934770:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504366848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763551.934804:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763551.934820:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763551/real 1789763551] req@ffff9018c66a4380 x1876699504366848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763551.934836:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504366848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763551.934847:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763551.934851:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763551.934856:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504366848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763551.934863:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504366976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763551.934907:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763551.934921:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763551/real 1789763551] req@ffff9018c66a7100 x1876699504366976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763551.934936:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504366976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763551.934946:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763551.934950:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763551.934961:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504366976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763551.934969:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504367104:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763551.934990:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763551.935003:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763551/real 1789763551] req@ffff9018c66a4a80 x1876699504367104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763551.935017:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504367104/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763551.935083:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763551.935088:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763551.935093:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504367104:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763551.935101:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504367232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763551.935130:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763551.935146:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763551/real 1789763551] req@ffff9018c66a7480 x1876699504367232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763551.935165:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504367232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763551.935175:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763551.935178:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763551.935182:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504367232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763551.935189:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504367360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763551.935207:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763551.935218:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763551/real 1789763551] req@ffff9018c66a5880 x1876699504367360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763551.935232:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504367360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763551.935241:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763551.935244:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763551.935248:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504367360:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763552.574293:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763553.598266:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763554.622264:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763555.646319:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763556.670248:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763557.054225:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763557.054242:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763557.054248:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763557.054264:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3483 00000100:00080000:2.0:1789763557.054282:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763557.054293:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763557.054357:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504367488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763557.054404:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763557.054409:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763557.054412:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763557.054434:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3483 00000100:00080000:2.0:1789763557.054439:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763557.054444:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763557.054471:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763557.054489:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504367616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763557.054503:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763557.054507:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763557.054510:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763557.054519:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3483 00000100:00080000:2.0:1789763557.054525:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763557.054527:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504367488:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763557.054535:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763557.054547:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763557.054571:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504367744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763557.054583:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763557.054588:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763557.054591:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763557.054596:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3483 00000100:00080000:2.0:1789763557.054602:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763557.054608:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763557.054614:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763557.054628:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504367872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763557.054642:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763557.054646:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763557.054648:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763557.054652:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3483 00000100:00080000:2.0:1789763557.054657:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763557.054661:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763557.054667:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763557.054681:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504368000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763557.054695:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763557.054703:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763557.054707:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763557.054720:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3483 00000100:00100000:0.0:1789763557.054722:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504367616:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763557.054736:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763557.054746:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763557.054755:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763557.054781:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504368128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763557.054807:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504367744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763557.054850:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504367872:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763557.054877:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504368000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763557.054910:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504368128:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763557.694173:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763558.718286:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763559.742254:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763560.126650:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:1.0:1789763560.126725:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789763560.126763:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763560.126870:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763560.126886:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763560.126910:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763560.126925:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763560.126939:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789763560.127025:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763557/real 1789763560] req@ffff9018d0f24380 x1876699504368128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763560.127107:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504368128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763560.127128:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763560.127137:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763560.127155:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504368128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763560.127170:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763557/real 1789763560] req@ffff9018d0f26d80 x1876699504367616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763560.127195:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504367616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763560.127210:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763560.127216:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763560.127222:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504367616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763560.127236:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763557/real 1789763560] req@ffff9018d0f25f80 x1876699504367744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763560.127256:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504367744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763560.127268:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763560.127273:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763560.127279:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504367744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763560.127289:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763557/real 1789763560] req@ffff9018d0f27100 x1876699504367872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763560.127307:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504367872/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763560.127319:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763560.127324:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763560.127328:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504367872:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763560.127338:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763557/real 1789763560] req@ffff9018d0f25180 x1876699504368000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763560.127356:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504368000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763560.127369:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763560.127373:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763560.127378:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504368000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763560.127390:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763557/real 1789763560] req@ffff9018d0f26300 x1876699504367488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763560.127412:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504367488/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763560.127425:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763560.127430:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763560.127436:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504367488:192.168.203.158@tcp:250:kworker.0 00010000:00010000:0.0:1789763560.766363:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763561.790335:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763562.174199:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763562.174215:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763562.174222:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763562.174238:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3488 00000100:00080000:2.0:1789763562.174255:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763562.174267:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763562.174334:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504368256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763562.174379:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763562.174387:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763562.174390:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763562.174399:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3488 00000100:00080000:2.0:1789763562.174407:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763562.174415:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763562.174444:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763562.174464:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504368256:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763562.174468:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504368384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763562.174489:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763562.174494:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763562.174498:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763562.174504:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3488 00000100:00080000:2.0:1789763562.174521:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763562.174529:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763562.174537:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763562.174555:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763562.174575:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504368512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763562.174593:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763562.174598:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763562.174602:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763562.174609:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3488 00000100:00100000:0.0:1789763562.174614:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763562/real 1789763562] req@ffff9018d0f24a80 x1876699504368256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763562.174615:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763562.174623:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763562.174632:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763562.174656:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504368256/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763562.174660:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504368640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763562.174679:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763562.174681:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763562.174686:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763562.174687:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763562.174690:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763562.174694:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504368256:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763562.174697:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3488 00000100:00080000:2.0:1789763562.174704:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763562.174710:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763562.174717:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763562.174732:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504368384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763562.174734:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504368768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763562.174763:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763562.174769:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763562.174772:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763562.174779:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3488 00000800:00000100:0.0:1789763562.174781:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763562.174785:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763562.174792:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763562.174800:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763562.174804:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763562/real 1789763562] req@ffff9018d0f26a00 x1876699504368384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763562.174818:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504368896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763562.174828:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504368384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763562.174846:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763562.174852:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763562.174859:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504368384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763562.174871:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504368512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763562.174912:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763562.174937:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763562/real 1789763562] req@ffff9018d0f25c00 x1876699504368512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763562.174999:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504368512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763562.175016:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763562.175022:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763562.175029:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504368512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763562.175090:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504368640:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763562.175133:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763562.175166:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763562/real 1789763562] req@ffff9018d0f27800 x1876699504368640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763562.175188:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504368640/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763562.175203:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763562.175208:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763562.175215:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504368640:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763562.175246:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504368768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763562.175308:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763562.175339:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763562/real 1789763562] req@ffff9018d0f24000 x1876699504368768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763562.175361:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504368768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763562.175377:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763562.175383:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763562.175389:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504368768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763562.175401:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504368896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763562.175435:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763562.175455:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763562/real 1789763562] req@ffff9018d0f26680 x1876699504368896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763562.175476:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504368896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763562.175488:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763562.175493:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763562.175498:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504368896:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763562.814266:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763563.838314:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763564.862324:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763565.886265:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763566.910292:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763567.294191:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763567.294204:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763567.294209:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763567.294223:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3493 00000100:00080000:1.0:1789763567.294240:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763567.294252:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763567.294311:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504369024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763567.294350:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763567.294356:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763567.294358:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763567.294364:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3493 00000100:00080000:1.0:1789763567.294371:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763567.294376:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763567.294390:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763567.294410:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504369152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763567.294424:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504369024:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763567.294430:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763567.294438:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763567.294440:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763567.294447:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3493 00000100:00080000:1.0:1789763567.294450:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763567.294456:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763567.294463:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763567.294482:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504369280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763567.294495:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763567.294499:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763567.294503:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763567.294508:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3493 00000100:00080000:1.0:1789763567.294513:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763567.294523:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763567.294524:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763567.294538:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763567.294554:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504369408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763567.294578:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763567.294582:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763567.294585:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763567.294589:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3493 00000100:00080000:1.0:1789763567.294595:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763567.294606:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763567.294607:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763567/real 1789763567] req@ffff9018c66a4e00 x1876699504369024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789763567.294612:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763567.294638:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504369536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763567.294652:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504369024/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763567.294658:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763567.294675:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763567.294683:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763567.294688:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763567.294693:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3493 00000100:00080000:0.0:1789763567.294701:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789763567.294705:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763567.294715:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504369024:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763567.294717:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763567.294734:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763567.294758:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504369664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763567.294769:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504369152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763567.294819:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763567.294841:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763567/real 1789763567] req@ffff9018c66a5500 x1876699504369152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763567.294858:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504369152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763567.294869:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763567.294873:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763567.294878:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504369152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763567.294885:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504369280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763567.294904:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763567.294949:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763567/real 1789763567] req@ffff9018c66a6680 x1876699504369280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763567.294975:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504369280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763567.294985:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763567.294989:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763567.294993:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504369280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763567.295000:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504369408:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763567.295027:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763567.295118:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763567/real 1789763567] req@ffff9018c66a4700 x1876699504369408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763567.295136:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504369408/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763567.295147:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763567.295150:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763567.295155:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504369408:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763567.295162:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504369536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763567.295187:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763567.295201:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763567/real 1789763567] req@ffff9018c66a6300 x1876699504369536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763567.295215:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504369536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763567.295233:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763567.295236:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763567.295240:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504369536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763567.295263:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504369664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763567.295283:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763567.295294:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763567/real 1789763567] req@ffff9018c66a5180 x1876699504369664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763567.295308:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504369664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763567.295318:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763567.295321:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763567.295325:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504369664:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763567.934229:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763568.958260:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763569.982307:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763571.006195:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763572.030237:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763572.414141:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763572.414155:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763572.414161:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763572.414189:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3498 00000100:00080000:1.0:1789763572.414205:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763572.414215:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763572.414270:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504369792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763572.414313:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763572.414321:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763572.414325:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763572.414335:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3498 00000100:00080000:1.0:1789763572.414345:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763572.414356:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763572.414373:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763572.414397:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504369792:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763572.414412:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504369920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763572.414440:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763572.414448:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763572.414453:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763572.414462:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3498 00000100:00080000:1.0:1789763572.414471:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763572.414479:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763572.414490:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763572.414492:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763572.414544:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504370048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763572.414568:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763572.414576:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763572.414580:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763572.414589:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3498 00000100:00080000:1.0:1789763572.414596:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763572.414604:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763572.414617:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763572.414644:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504370176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763572.414668:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763572.414675:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763572.414681:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763572.414691:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3498 00000100:00080000:1.0:1789763572.414699:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763572.414706:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763572.414716:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763572.414743:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504370304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763572.414758:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763572.414763:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763572.414765:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763572.414770:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3498 00000100:00080000:1.0:1789763572.414774:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763572.414779:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763572.414786:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763572.414815:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504370432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763572.414866:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763572/real 1789763572] req@ffff9018c66a7800 x1876699504369792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763572.414943:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504369792/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763572.414958:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763572.414964:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763572.414972:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504369792:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763572.415222:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504369920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763572.415262:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763572.415283:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763572/real 1789763572] req@ffff9018c66a7b80 x1876699504369920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763572.415300:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504369920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763572.415319:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763572.415324:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763572.415328:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504369920:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763572.415336:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504370048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763572.415357:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763572.415369:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763572/real 1789763572] req@ffff9018c66a4000 x1876699504370048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763572.415383:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504370048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763572.415393:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763572.415396:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763572.415400:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504370048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763572.415406:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504370176:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763572.415426:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763572.415438:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763572/real 1789763572] req@ffff9018c66a5f80 x1876699504370176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763572.415452:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504370176/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763572.415461:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763572.415464:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763572.415469:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504370176:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763572.415476:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504370304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763572.415495:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763572.415506:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763572/real 1789763572] req@ffff9018c66a6d80 x1876699504370304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763572.415520:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504370304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763572.415529:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763572.415533:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763572.415536:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504370304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763572.415543:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504370432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763572.415560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763572.415571:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763572/real 1789763572] req@ffff9018c66a5c00 x1876699504370432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763572.415584:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504370432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763572.415593:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763572.415597:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763572.415601:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504370432:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763573.054327:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763574.078277:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763575.102249:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763576.126261:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763577.150260:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763577.534166:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763577.534179:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763577.534184:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763577.534197:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3504 00000100:00080000:2.0:1789763577.534211:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763577.534220:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763577.534267:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504370560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763577.534305:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763577.534311:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763577.534327:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763577.534334:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3504 00000100:00080000:2.0:1789763577.534340:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763577.534346:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763577.534361:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763577.534376:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504370560:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763577.534385:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504370688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763577.534404:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763577.534409:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763577.534412:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763577.534416:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3504 00000100:00080000:2.0:1789763577.534422:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763577.534426:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763577.534433:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763577.534456:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504370816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763577.534478:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763577.534496:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763577.534504:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763577.534509:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763577.534520:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3504 00000100:00080000:2.0:1789763577.534530:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763577.534540:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763577.534554:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763577/real 1789763577] req@ffff9018d0f24700 x1876699504370560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763577.534555:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763577.534606:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504370560/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763577.534608:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504370944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763577.534674:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763577.534678:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763577.534684:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763577.534690:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1789763577.534697:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504370560:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763577.534699:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763577.534717:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3504 00000100:00080000:2.0:1789763577.534724:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763577.534730:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763577.534738:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763577.534748:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504370688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763577.534762:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504371072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763577.534788:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789763577.534795:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763577.534807:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763577.534821:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763577.534829:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763577/real 1789763577] req@ffff9018d0f25500 x1876699504370688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763577.534859:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3504 00000100:00080000:2.0:1789763577.534876:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763577.534890:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504370688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763577.534907:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789763577.534917:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789763577.534922:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789763577.534928:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763577.534937:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504370688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763577.534948:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504370816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763577.534955:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504371200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763577.534991:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763577.535014:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763577/real 1789763577] req@ffff9018d0f27480 x1876699504370816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763577.535070:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504370816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763577.535083:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763577.535087:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763577.535092:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504370816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763577.535100:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504370944:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763577.535152:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763577.535168:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763577/real 1789763577] req@ffff9018d0f24e00 x1876699504370944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763577.535184:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504370944/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763577.535194:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763577.535199:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763577.535202:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504370944:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763577.535223:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504371072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763577.535245:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763577.535258:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763577/real 1789763577] req@ffff9018d0f24700 x1876699504371072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763577.535272:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504371072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763577.535291:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763577.535295:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763577.535299:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504371072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763577.535307:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504371200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763577.535340:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763577.535361:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763577/real 1789763577] req@ffff9018d0f26680 x1876699504371200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763577.535386:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504371200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763577.535403:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763577.535409:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763577.535416:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504371200:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763578.174226:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763579.198267:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763580.222231:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763581.246265:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763582.270244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763582.654178:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763582.654195:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763582.654202:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763582.654235:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3509 00000100:00080000:2.0:1789763582.654252:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763582.654264:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763582.654321:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504371328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763582.654358:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763582.654363:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763582.654366:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763582.654371:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3509 00000100:00080000:2.0:1789763582.654376:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763582.654381:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763582.654394:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763582.654412:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504371456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763582.654424:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763582.654429:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763582.654431:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763582.654436:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3509 00000100:00080000:2.0:1789763582.654440:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763582.654444:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763582.654451:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763582.654462:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504371328:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763582.654467:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504371584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763582.654507:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763582.654514:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763582.654518:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763582.654527:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3509 00000100:00080000:2.0:1789763582.654562:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763582.654571:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:3.0:1789763582.654582:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789763582.654587:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763582.654613:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504371712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763582.654627:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763582.654633:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763582.654635:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763582.654641:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3509 00000100:00080000:2.0:1789763582.654648:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763582.654658:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763582.654659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763582/real 1789763582] req@ffff9018d0f24000 x1876699504371328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763582.654667:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763582.654691:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504371840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763582.654705:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504371328/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763582.654711:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763582.654729:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763582.654736:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763582.654750:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3509 00000100:00080000:2.0:1789763582.654762:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763582.654771:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:3.0:1789763582.654775:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789763582.654777:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763582.654824:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504371968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763582.654827:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763582.654851:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504371328:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763582.654871:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504371456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763582.654922:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763582.654951:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763582/real 1789763582] req@ffff9018d0f27800 x1876699504371456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763582.654980:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504371456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763582.654997:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763582.655002:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763582.655007:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504371456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763582.655083:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504371584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763582.655115:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763582.655131:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763582/real 1789763582] req@ffff9018d0f25c00 x1876699504371584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763582.655147:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504371584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763582.655157:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763582.655161:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763582.655165:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504371584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763582.655174:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504371712:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763582.655194:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763582.655206:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763582/real 1789763582] req@ffff9018d0f26a00 x1876699504371712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763582.655220:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504371712/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763582.655230:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763582.655233:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763582.655237:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504371712:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763582.655243:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504371840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763582.655262:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763582.655273:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763582/real 1789763582] req@ffff9018d0f24a80 x1876699504371840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763582.655287:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504371840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763582.655296:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763582.655300:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763582.655303:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504371840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763582.655310:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504371968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763582.655328:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763582.655339:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763582/real 1789763582] req@ffff9018d0f26300 x1876699504371968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763582.655353:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504371968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763582.655363:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763582.655366:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763582.655370:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504371968:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763583.294318:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763584.318308:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763585.342189:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763586.366183:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763587.390285:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763587.774261:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763587.774286:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763587.774291:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763587.774304:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3514 00000100:00080000:1.0:1789763587.774319:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763587.774328:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763587.774397:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504372096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763587.774453:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763587.774457:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763587.774459:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763587.774465:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3514 00000100:00080000:1.0:1789763587.774469:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763587.774474:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763587.774486:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763587.774503:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504372224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763587.774515:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763587.774519:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763587.774521:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763587.774525:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3514 00000100:00080000:1.0:1789763587.774531:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763587.774535:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504372096:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763587.774537:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763587.774546:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763587.774556:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504372352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763587.774565:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763587.774568:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763587.774570:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763587.774573:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3514 00000100:00080000:1.0:1789763587.774576:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763587.774579:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763587.774584:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763587.774601:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504372480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763587.774610:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763587.774613:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763587.774615:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763587.774618:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3514 00000100:00080000:1.0:1789763587.774621:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763587.774624:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763587.774628:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763587.774637:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504372608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763587.774661:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763587.774665:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763587.774666:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763587.774669:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3514 00000100:00080000:1.0:1789763587.774673:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789763587.774676:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763587.774681:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763587.774687:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763587.774697:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504372736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763587.774742:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763587/real 1789763587] req@ffff9018c66a5c00 x1876699504372096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763587.774790:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504372096/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763587.774813:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763587.774817:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763587.774821:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504372096:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763587.774852:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504372224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763587.774877:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763587.774889:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763587/real 1789763587] req@ffff9018c66a6d80 x1876699504372224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763587.774906:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504372224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763587.774914:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763587.774927:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763587.774930:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504372224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763587.774935:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504372352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763587.774949:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763587.774957:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763587/real 1789763587] req@ffff9018c66a5f80 x1876699504372352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763587.774967:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504372352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763587.774974:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763587.774976:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763587.774978:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504372352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763587.774983:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504372480:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763587.774997:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763587.775004:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763587/real 1789763587] req@ffff9018c66a4000 x1876699504372480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763587.775014:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504372480/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763587.775027:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763587.775030:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763587.775071:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504372480:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763587.775080:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504372608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763587.775109:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763587.775127:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763587/real 1789763587] req@ffff9018c66a7b80 x1876699504372608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763587.775143:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504372608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763587.775155:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763587.775159:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763587.775164:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504372608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763587.775172:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504372736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763587.775202:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763587.775217:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763587/real 1789763587] req@ffff9018c66a7800 x1876699504372736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763587.775228:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504372736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763587.775235:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763587.775238:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763587.775242:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504372736:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763588.414315:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763589.438171:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763590.462237:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763591.486336:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763592.510271:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763592.894143:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763592.894153:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763592.894157:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763592.894166:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3519 00000100:00080000:2.0:1789763592.894176:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763592.894184:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763592.894218:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504372864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763592.894239:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763592.894241:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763592.894243:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763592.894246:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3519 00000100:00080000:2.0:1789763592.894248:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763592.894251:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763592.894259:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763592.894267:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504372992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763592.894273:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763592.894276:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763592.894278:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763592.894280:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3519 00000100:00080000:2.0:1789763592.894282:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763592.894285:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763592.894289:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763592.894297:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504373120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763592.894304:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763592.894306:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763592.894308:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763592.894311:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3519 00000100:00080000:2.0:1789763592.894314:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763592.894317:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763592.894320:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763592.894329:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504373248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763592.894329:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504372864:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763592.894352:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763592.894354:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763592.894356:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763592.894361:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3519 00000100:00080000:2.0:1789763592.894363:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763592.894367:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763592.894371:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763592.894378:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504373376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763592.894386:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763592.894388:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763592.894389:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763592.894394:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3519 00000100:00080000:2.0:1789763592.894411:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763592.894413:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763592.894418:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763592.894422:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763592.894429:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504373504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763592.894479:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763592/real 1789763592] req@ffff9018d0f25180 x1876699504372864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763592.894498:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504372864/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763592.894508:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763592.894512:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763592.894515:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504372864:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763592.894521:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504372992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763592.894537:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763592.894556:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763592/real 1789763592] req@ffff9018d0f27100 x1876699504372992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763592.894564:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504372992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763592.894570:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763592.894572:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763592.894574:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504372992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763592.894580:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504373120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763592.894592:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763592.894598:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763592/real 1789763592] req@ffff9018d0f25f80 x1876699504373120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763592.894606:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504373120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763592.894621:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763592.894622:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763592.894625:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504373120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763592.894652:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504373248:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763592.894664:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763592.894670:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763592/real 1789763592] req@ffff9018d0f26d80 x1876699504373248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763592.894677:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504373248/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763592.894682:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763592.894684:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763592.894686:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504373248:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763592.894691:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504373376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763592.894701:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763592.894707:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763592/real 1789763592] req@ffff9018d0f24380 x1876699504373376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763592.894715:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504373376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763592.894735:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763592.894738:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763592.894740:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504373376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763592.894744:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504373504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763592.894757:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763592.894764:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763592/real 1789763592] req@ffff9018d0f26300 x1876699504373504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763592.894781:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504373504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763592.894786:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763592.894788:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763592.894790:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504373504:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763593.534191:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763594.558248:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763595.582229:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763596.606232:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763597.630342:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763598.014203:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763598.014231:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763598.014236:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763598.014249:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3524 00000100:00080000:3.0:1789763598.014264:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763598.014273:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763598.014329:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504373632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763598.014368:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763598.014373:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763598.014375:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763598.014381:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3524 00000100:00080000:3.0:1789763598.014385:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763598.014389:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763598.014402:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763598.014417:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504373760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763598.014429:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763598.014433:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763598.014436:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763598.014441:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3524 00000100:00080000:3.0:1789763598.014445:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763598.014450:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763598.014454:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763598.014466:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504373888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763598.014479:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763598.014486:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504373632:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763598.014487:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763598.014495:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763598.014509:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3524 00000100:00080000:3.0:1789763598.014515:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763598.014520:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763598.014527:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763598.014543:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504374016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763598.014556:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763598.014560:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763598.014563:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763598.014568:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3524 00000100:00080000:3.0:1789763598.014573:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763598.014578:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763598.014584:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763598.014602:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763598.014603:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504374144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763598.014621:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763598.014625:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763598.014629:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763598.014635:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3524 00000100:00080000:3.0:1789763598.014639:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763598.014643:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763598.014649:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763598.014673:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763598/real 1789763598] req@ffff9018d070e680 x1876699504373632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763598.014674:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504374272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763598.014749:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504373632/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763598.014765:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763598.014771:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763598.014777:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504373632:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763598.014787:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504373760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763598.014815:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763598.014831:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763598/real 1789763598] req@ffff9018d070d180 x1876699504373760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763598.014846:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504373760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763598.014857:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763598.014861:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763598.014865:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504373760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763598.014894:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504373888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763598.014914:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763598.014926:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763598/real 1789763598] req@ffff9018d070ca80 x1876699504373888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763598.014949:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504373888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763598.014958:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763598.014962:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763598.014965:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504373888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763598.014972:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504374016:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763598.014990:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763598.015001:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763598/real 1789763598] req@ffff9018d070d880 x1876699504374016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763598.015015:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504374016/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763598.015025:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763598.015028:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763598.015070:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504374016:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763598.015079:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504374144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763598.015102:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763598.015116:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763598/real 1789763598] req@ffff9018d070ed80 x1876699504374144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763598.015130:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504374144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763598.015148:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763598.015152:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763598.015156:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504374144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763598.015162:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504374272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763598.015204:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763598.015217:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763598/real 1789763598] req@ffff9018d070ce00 x1876699504374272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763598.015231:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504374272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763598.015241:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763598.015244:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763598.015248:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504374272:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763598.654276:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763599.678213:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763600.702277:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763601.726342:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763602.750233:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763603.134169:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763603.134186:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763603.134193:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763603.134206:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3529 00000100:00080000:0.0:1789763603.134219:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763603.134229:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763603.134272:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504374400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763603.134328:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763603.134337:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763603.134340:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763603.134349:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3529 00000100:00080000:0.0:1789763603.134358:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763603.134368:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763603.134386:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763603.134387:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504374400:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763603.134425:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504374528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763603.134454:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763603.134459:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763603.134461:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763603.134467:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3529 00000100:00080000:0.0:1789763603.134473:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763603.134479:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763603.134486:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763603.134495:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763603.134509:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504374656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763603.134525:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763603.134529:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763603.134532:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763603.134538:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3529 00000100:00080000:0.0:1789763603.134545:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763603.134550:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763603.134558:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763603/real 1789763603] req@ffff9018c8b98a80 x1876699504374400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763603.134563:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763603.134590:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504374784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763603.134601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504374400/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763603.134616:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763603.134630:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789763603.134639:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763603.134670:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504374400:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763603.134672:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763603.134677:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763603.134695:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3529 00000100:00080000:0.0:1789763603.134703:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763603.134709:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763603.134717:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763603.134737:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504374528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763603.134740:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504374912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763603.134767:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763603.134772:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763603.134775:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763603.134782:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3529 00000100:00080000:0.0:1789763603.134787:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763603.134793:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763603.134801:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763603.134804:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763603.134826:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504375040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763603.134839:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763603/real 1789763603] req@ffff9018c8b99f80 x1876699504374528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763603.134865:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504374528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763603.134878:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763603.134883:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763603.134889:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504374528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763603.134899:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504374656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763603.134923:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763603.134935:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763603/real 1789763603] req@ffff9018c8b98700 x1876699504374656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763603.134950:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504374656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763603.134960:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763603.134963:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763603.134968:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504374656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763603.134975:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504374784:192.168.203.158@tcp:8:kworker.0 00000800:00000100:3.0:1789763603.134994:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763603.135005:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763603/real 1789763603] req@ffff9018c8b9bb80 x1876699504374784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763603.135019:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504374784/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763603.135068:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763603.135073:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763603.135077:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504374784:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763603.135100:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504374912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763603.135134:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763603.135158:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763603/real 1789763603] req@ffff9018c8b9b100 x1876699504374912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763603.135183:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504374912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763603.135201:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763603.135207:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763603.135214:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504374912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763603.135226:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504375040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:3.0:1789763603.135262:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763603.135277:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763603/real 1789763603] req@ffff9018c8b9b480 x1876699504375040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763603.135293:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504375040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763603.135303:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763603.135307:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:3.0:1789763603.135310:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504375040:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763603.774288:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763604.798258:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763605.822264:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763606.846270:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763607.870257:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763608.254195:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763608.254211:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763608.254218:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763608.254237:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3534 00000100:00080000:0.0:1789763608.254255:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763608.254268:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763608.254326:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504375168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763608.254381:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763608.254390:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763608.254394:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763608.254406:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3534 00000100:00080000:0.0:1789763608.254418:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763608.254429:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763608.254448:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763608.254464:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504375168:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763608.254480:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504375296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763608.254501:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763608.254507:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763608.254509:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763608.254515:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3534 00000100:00080000:0.0:1789763608.254521:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763608.254526:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763608.254532:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763608.254547:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504375424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763608.254561:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763608.254565:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763608.254568:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763608.254577:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763608.254579:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3534 00000100:00080000:0.0:1789763608.254591:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763608.254597:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763608.254604:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763608.254652:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504375552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763608.254669:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763608.254677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763608/real 1789763608] req@ffff9018c8b9b480 x1876699504375168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763608.254678:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763608.254686:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763608.254701:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3534 00000100:00080000:0.0:1789763608.254714:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763608.254721:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763608.254729:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504375168/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763608.254733:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763608.254758:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789763608.254760:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504375680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763608.254770:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763608.254787:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504375168:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763608.254793:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763608.254800:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763608.254804:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763608.254811:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3534 00000100:00080000:0.0:1789763608.254815:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763608.254821:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763608.254830:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763608.254843:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504375296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763608.254858:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504375808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763608.254891:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763608.254917:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763608/real 1789763608] req@ffff9018c8b9b100 x1876699504375296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763608.254939:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504375296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763608.254953:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763608.254960:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763608.254965:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504375296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763608.254989:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504375424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763608.255024:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763608.255082:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763608/real 1789763608] req@ffff9018c8b9bb80 x1876699504375424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763608.255109:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504375424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763608.255122:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763608.255130:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763608.255139:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504375424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763608.255152:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504375552:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763608.255185:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763608.255205:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763608/real 1789763608] req@ffff9018c8b98700 x1876699504375552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763608.255220:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504375552/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763608.255240:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763608.255244:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763608.255248:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504375552:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763608.255254:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504375680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763608.255288:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763608.255301:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763608/real 1789763608] req@ffff9018c8b99f80 x1876699504375680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763608.255315:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504375680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763608.255325:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763608.255329:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763608.255333:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504375680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763608.255354:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504375808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763608.255375:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763608.255387:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763608/real 1789763608] req@ffff9018c8b98a80 x1876699504375808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763608.255401:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504375808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763608.255410:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763608.255413:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763608.255418:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504375808:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763608.894273:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763609.918291:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763610.942250:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763611.966230:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763612.990243:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763613.374162:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763613.374185:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763613.374190:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763613.374204:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3539 00000100:00080000:0.0:1789763613.374220:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763613.374233:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763613.374288:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504375936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763613.374337:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763613.374344:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763613.374348:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763613.374356:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3539 00000100:00080000:0.0:1789763613.374365:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763613.374375:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763613.374393:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763613.374404:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504375936:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763613.374423:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504376064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763613.374442:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763613.374447:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763613.374450:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763613.374455:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3539 00000100:00080000:0.0:1789763613.374474:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763613.374480:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763613.374487:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763613.374490:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763613.374508:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504376192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763613.374522:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763613.374527:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763613.374529:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763613.374535:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3539 00000100:00080000:0.0:1789763613.374539:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763613.374543:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763613.374551:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763613.374559:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763613/real 1789763613] req@ffff9018c8b9a680 x1876699504375936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763613.374595:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504376320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763613.374615:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763613.374619:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504375936/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763613.374621:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763613.374629:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763613.374646:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3539 00000100:00080000:1.0:1789763613.374654:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763613.374655:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763613.374669:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763613.374672:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:0.0:1789763613.374680:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763613.374683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504375936:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763613.374706:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504376448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763613.374723:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763613.374727:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504376064:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763613.374730:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763613.374736:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763613.374747:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3539 00000100:00080000:0.0:1789763613.374753:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763613.374757:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763613.374764:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763613.374780:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763613.374783:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504376576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763613.374807:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763613/real 1789763613] req@ffff9018c8b98e00 x1876699504376064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.374831:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504376064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.374843:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.374847:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763613.374851:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504376064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763613.374859:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504376192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763613.374881:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763613.374894:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763613/real 1789763613] req@ffff9018c8b9aa00 x1876699504376192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.374908:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504376192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.374918:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.374922:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763613.374926:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504376192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763613.374933:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504376320:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763613.374953:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763613.374977:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763613/real 1789763613] req@ffff9018c8b9b800 x1876699504376320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.374993:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504376320/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.375003:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.375007:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763613.375011:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504376320:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763613.375030:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504376448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763613.375090:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763613.375105:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763613/real 1789763613] req@ffff9018c8b98000 x1876699504376448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.375120:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504376448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.375130:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.375135:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763613.375139:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504376448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763613.375145:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504376576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763613.375164:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763613.375174:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763613/real 1789763613] req@ffff9018c8b99880 x1876699504376576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.375188:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504376576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.375197:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.375201:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763613.375204:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504376576:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763614.014311:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763615.038248:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763616.062278:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763617.086201:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763618.110256:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763618.494144:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763618.494157:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763618.494162:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763618.494175:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3545 00000100:00080000:0.0:1789763618.494188:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763618.494198:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763618.494246:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504376704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763618.494283:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763618.494289:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763618.494292:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763618.494296:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3545 00000100:00080000:0.0:1789763618.494300:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763618.494305:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763618.494317:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763618.494334:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504376832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763618.494350:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763618.494358:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763618.494362:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763618.494362:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504376704:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763618.494377:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3545 00000100:00080000:0.0:1789763618.494382:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763618.494386:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763618.494392:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763618.494408:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504376960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763618.494419:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763618.494424:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763618.494427:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763618.494432:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3545 00000100:00080000:0.0:1789763618.494441:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763618.494446:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763618.494453:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763618.494461:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763618.494469:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504377088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763618.494485:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763618.494489:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763618.494491:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763618.494496:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3545 00000100:00080000:0.0:1789763618.494500:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763618.494505:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763618.494510:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763618.494521:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504377216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763618.494551:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763618/real 1789763618] req@ffff9018c8b99c00 x1876699504376704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763618.494564:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763618.494577:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763618.494601:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763618.494608:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504376704/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763618.494611:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3545 00000100:00080000:0.0:1789763618.494622:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763618.494634:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763618.494656:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763618.494657:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763618.494669:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763618.494679:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504377344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763618.494681:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504376704:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763618.494726:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504376832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763618.494772:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763618.494788:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763618/real 1789763618] req@ffff9018c8b9ad80 x1876699504376832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763618.494804:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504376832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763618.494814:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763618.494818:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763618.494823:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504376832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763618.494830:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504376960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763618.494849:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763618.494860:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763618/real 1789763618] req@ffff9018c8b9a300 x1876699504376960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763618.494874:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504376960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763618.494884:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763618.494887:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763618.494891:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504376960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763618.494897:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504377088:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763618.494916:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763618.494928:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763618/real 1789763618] req@ffff9018c8b98380 x1876699504377088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763618.494942:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504377088/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763618.494951:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763618.494955:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763618.494958:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504377088:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763618.494964:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504377216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763618.494982:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763618.494993:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763618/real 1789763618] req@ffff9018c8b99500 x1876699504377216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763618.495007:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504377216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763618.495016:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763618.495019:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763618.495023:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504377216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763618.495029:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504377344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763618.495097:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763618.495112:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763618/real 1789763618] req@ffff9018c8b99880 x1876699504377344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763618.495126:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504377344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763618.495135:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763618.495139:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763618.495143:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504377344:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763619.134252:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763620.158238:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763621.182230:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763622.206279:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763623.230297:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763623.614255:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763623.614267:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763623.614273:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763623.614285:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3550 00000100:00080000:2.0:1789763623.614314:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763623.614324:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763623.614374:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504377472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763623.614410:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763623.614416:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763623.614418:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763623.614423:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3550 00000100:00080000:2.0:1789763623.614428:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763623.614433:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763623.614456:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763623.614481:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504377600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763623.614520:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763623.614523:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504377472:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763623.614534:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763623.614543:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763623.614550:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3550 00000100:00080000:2.0:1789763623.614555:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763623.614560:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763623.614568:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763623.614590:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504377728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763623.614603:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763623.614608:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763623.614611:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763623.614616:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3550 00000100:00080000:2.0:1789763623.614621:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763623.614626:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763623.614632:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763623.614645:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504377856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763623.614667:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763623.614671:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763623.614674:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763623.614679:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3550 00000100:00080000:2.0:1789763623.614682:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763623.614688:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763623.614695:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763623.614699:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504377600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763623.614716:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504377984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763623.614736:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763623.614740:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763623.614743:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763623.614750:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3550 00000100:00100000:1.0:1789763623.614757:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504377728:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763623.614758:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763623.614771:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763623.614781:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763623.614798:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504378112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763623.614816:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504377856:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763623.614888:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504377984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763623.614935:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504378112:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763624.254262:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763625.278273:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763626.302262:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763626.686384:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:0.0:1789763626.686441:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763626.686462:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763626.686543:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763626.686555:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763626.686565:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763626.686579:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763626.686591:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789763626.686632:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763623/real 1789763626] req@ffff9018d0f24000 x1876699504377984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763626.686682:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504377984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763626.686703:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763626.686712:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763626.686720:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504377984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763626.686735:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763623/real 1789763626] req@ffff9018d0f26680 x1876699504378112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763626.686786:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504378112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763626.686799:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763626.686806:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763626.686811:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504378112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763626.686822:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763623/real 1789763626] req@ffff9018d0f26a00 x1876699504377600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763626.686841:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504377600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763626.686854:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763626.686859:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763626.686865:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504377600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763626.686876:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763623/real 1789763626] req@ffff9018d0f25c00 x1876699504377728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763626.686896:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504377728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763626.686910:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763626.686916:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763626.686922:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504377728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763626.686933:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763623/real 1789763626] req@ffff9018d0f27800 x1876699504377856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763626.686957:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504377856/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763626.686973:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763626.686989:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763626.686996:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504377856:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763626.687009:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763623/real 1789763626] req@ffff9018d0f24a80 x1876699504377472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763626.687033:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504377472/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763626.687096:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763626.687103:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763626.687109:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504377472:192.168.203.158@tcp:250:kworker.0 00010000:00010000:3.0:1789763627.326281:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763628.350301:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763628.734167:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763628.734184:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763628.734191:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763628.734206:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3555 00000100:00080000:2.0:1789763628.734220:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763628.734230:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763628.734275:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504378240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763628.734324:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763628.734329:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763628.734332:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763628.734337:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3555 00000100:00080000:2.0:1789763628.734341:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763628.734346:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763628.734358:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763628.734374:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504378368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763628.734392:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763628.734399:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763628.734403:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763628.734412:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3555 00000100:00100000:1.0:1789763628.734419:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504378240:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763628.734423:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763628.734441:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763628.734454:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763628.734525:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504378496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763628.734544:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763628.734568:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763628.734573:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763628.734582:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3555 00000100:00080000:2.0:1789763628.734591:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763628.734599:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763628.734608:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763628.734621:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763628.734650:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504378624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763628.734669:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763628.734676:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763628.734682:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763628.734686:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763628/real 1789763628] req@ffff9018d0f24700 x1876699504378240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763628.734699:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3555 00000100:00080000:2.0:1789763628.734714:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763628.734729:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763628.734741:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763628.734744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504378240/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763628.734772:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789763628.734775:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504378752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763628.734786:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763628.734806:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504378240:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763628.734818:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763628.734830:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789763628.734832:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504378368:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763628.734839:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763628.734865:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3555 00000100:00080000:2.0:1789763628.734874:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763628.734883:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763628.734895:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763628.734907:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763628.734927:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504378880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763628.734946:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763628/real 1789763628] req@ffff9018d0f24e00 x1876699504378368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763628.734977:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504378368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763628.734997:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763628.735004:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763628.735011:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504378368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763628.735110:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504378496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763628.735159:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763628.735197:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763628/real 1789763628] req@ffff9018d0f27480 x1876699504378496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763628.735222:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504378496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763628.735239:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763628.735246:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763628.735252:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504378496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763628.735265:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504378624:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763628.735307:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763628.735330:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763628/real 1789763628] req@ffff9018d0f25500 x1876699504378624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763628.735355:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504378624/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763628.735372:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763628.735378:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763628.735384:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504378624:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763628.735407:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504378752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763628.735452:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763628.735505:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763628/real 1789763628] req@ffff9018d0f24a80 x1876699504378752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763628.735529:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504378752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763628.735544:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763628.735551:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763628.735557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504378752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763628.735567:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504378880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763628.735603:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763628.735622:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763628/real 1789763628] req@ffff9018d0f27800 x1876699504378880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763628.735642:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504378880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763628.735667:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763628.735672:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763628.735678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504378880:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763629.374243:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763630.398244:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763631.422243:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763632.446285:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763633.470271:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763633.854152:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763633.854165:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763633.854171:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763633.854185:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3560 00000100:00080000:2.0:1789763633.854201:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763633.854211:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763633.854264:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504379008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763633.854299:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763633.854303:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763633.854306:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763633.854311:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3560 00000100:00080000:2.0:1789763633.854316:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763633.854321:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763633.854334:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763633.854366:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504379136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763633.854386:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763633.854391:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504379008:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763633.854392:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763633.854401:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763633.854443:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3560 00000100:00080000:2.0:1789763633.854450:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763633.854455:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763633.854461:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763633.854484:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504379264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763633.854498:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763633.854503:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763633.854506:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763633.854511:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3560 00000100:00080000:2.0:1789763633.854520:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763633.854525:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763633.854531:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763633.854547:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504379392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763633.854559:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763633.854564:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763633.854566:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763633.854583:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3560 00000800:00000100:1.0:1789763633.854585:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763633.854590:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763633.854599:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763633.854604:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763633.854620:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504379520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763633.854652:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763633.854656:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763633.854659:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763633.854666:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3560 00000100:00080000:2.0:1789763633.854672:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763633.854679:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763633/real 1789763633] req@ffff9018d0f25c00 x1876699504379008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763633.854686:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763633.854698:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763633.854718:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504379648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763633.854761:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504379008/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763633.854783:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763633.854790:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763633.854795:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504379008:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763633.854844:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504379136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763633.854882:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763633.854900:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763633/real 1789763633] req@ffff9018d0f26a00 x1876699504379136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763633.854916:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504379136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763633.854928:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763633.854933:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763633.854937:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504379136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763633.854946:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504379264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763633.854969:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763633.854981:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763633/real 1789763633] req@ffff9018d0f26680 x1876699504379264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763633.855009:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504379264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763633.855019:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763633.855024:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763633.855028:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504379264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763633.855080:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504379392:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763633.855109:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763633.855125:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763633/real 1789763633] req@ffff9018d0f24000 x1876699504379392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763633.855140:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504379392/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763633.855151:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763633.855156:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763633.855160:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504379392:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763633.855168:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504379520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763633.855205:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763633.855218:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763633/real 1789763633] req@ffff9018d0f26300 x1876699504379520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763633.855234:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504379520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763633.855244:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763633.855248:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763633.855252:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504379520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763633.855259:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504379648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763633.855280:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763633.855292:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763633/real 1789763633] req@ffff9018d0f24380 x1876699504379648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763633.855306:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504379648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763633.855316:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763633.855319:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763633.855323:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504379648:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763634.494260:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763635.518264:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763636.542220:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763637.566320:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763638.590266:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763638.974217:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763638.974235:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763638.974242:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763638.974258:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3565 00000100:00080000:0.0:1789763638.974276:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763638.974288:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763638.974352:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504379776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763638.974421:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763638.974427:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763638.974430:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763638.974436:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3565 00000100:00080000:0.0:1789763638.974442:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763638.974449:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763638.974464:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763638.974488:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504379904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763638.974501:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504379776:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763638.974509:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763638.974516:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763638.974518:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763638.974524:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3565 00000100:00080000:0.0:1789763638.974528:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763638.974533:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763638.974539:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763638.974555:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504380032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763638.974568:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763638.974573:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763638.974575:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763638.974580:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3565 00000100:00080000:0.0:1789763638.974585:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763638.974590:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763638.974597:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763638.974608:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763638.974618:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504380160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763638.974635:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763638.974641:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763638.974644:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763638.974648:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3565 00000100:00080000:0.0:1789763638.974652:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763638.974656:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763638.974663:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763638.974677:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504380288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763638.974687:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763638.974691:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763638.974694:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763638.974703:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3565 00000100:00100000:1.0:1789763638.974706:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763638/real 1789763638] req@ffff9018c8b98000 x1876699504379776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763638.974708:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763638.974721:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763638.974734:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763638.974753:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504380416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763638.974763:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504379776/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763638.974795:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763638.974802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763638.974809:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504379776:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763638.974844:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504379904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763638.974891:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763638.974918:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763638/real 1789763638] req@ffff9018c8b9b800 x1876699504379904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763638.974944:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504379904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763638.974955:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763638.974960:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763638.974964:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504379904:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763638.974972:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504380032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763638.975000:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763638.975015:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763638/real 1789763638] req@ffff9018c8b9aa00 x1876699504380032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763638.975029:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504380032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763638.975082:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763638.975087:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763638.975091:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504380032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763638.975099:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504380160:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763638.975125:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763638.975140:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763638/real 1789763638] req@ffff9018c8b98e00 x1876699504380160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763638.975155:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504380160/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763638.975165:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763638.975169:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763638.975173:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504380160:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763638.975180:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504380288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763638.975213:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763638.975235:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763638/real 1789763638] req@ffff9018c8b9a680 x1876699504380288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763638.975259:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504380288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763638.975277:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763638.975283:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763638.975290:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504380288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763638.975300:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504380416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763638.975330:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763638.975345:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763638/real 1789763638] req@ffff9018c8b98a80 x1876699504380416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763638.975361:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504380416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763638.975371:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763638.975375:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763638.975378:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504380416:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763639.614260:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763640.638262:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763641.662296:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763642.686262:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763643.710307:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763644.094134:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763644.094151:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763644.094157:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763644.094173:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3570 00000100:00080000:0.0:1789763644.094191:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763644.094204:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763644.094282:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504380544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763644.094319:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763644.094326:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763644.094329:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763644.094336:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3570 00000100:00080000:0.0:1789763644.094341:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763644.094373:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763644.094392:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763644.094425:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504380544:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763644.094426:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504380672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763644.094462:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763644.094468:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763644.094472:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763644.094478:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3570 00000100:00080000:0.0:1789763644.094484:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763644.094489:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763644.094496:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763644.094515:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504380800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763644.094531:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763644.094540:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763644.094545:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763644.094557:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763644.094559:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3570 00000100:00080000:0.0:1789763644.094575:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763644.094587:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763644.094599:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763644.094633:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504380928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763644.094638:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763644/real 1789763644] req@ffff9018c8b99f80 x1876699504380544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763644.094686:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763644.094697:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763644.094702:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763644.094713:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504380544/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763644.094718:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3570 00000100:00080000:0.0:1789763644.094736:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763644.094747:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763644.094756:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763644.094763:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763644.094776:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504380544:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789763644.094777:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763644.094822:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504381056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763644.094847:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763644.094855:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763644.094860:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763644.094865:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504380672:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763644.094883:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3570 00000100:00080000:0.0:1789763644.094902:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763644.094915:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763644.094932:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763644.094945:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763644.094973:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504381184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763644.094996:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763644/real 1789763644] req@ffff9018c8b98700 x1876699504380672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763644.095029:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504380672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763644.095142:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763644.095152:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763644.095165:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504380672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763644.095187:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504380800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763644.095235:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763644.095254:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763644/real 1789763644] req@ffff9018c8b9bb80 x1876699504380800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763644.095270:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504380800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763644.095281:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763644.095285:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763644.095289:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504380800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763644.095297:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504380928:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763644.095319:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763644.095331:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763644/real 1789763644] req@ffff9018c8b9b100 x1876699504380928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763644.095373:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504380928/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763644.095385:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763644.095389:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763644.095395:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504380928:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763644.095403:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504381056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763644.095432:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763644.095447:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763644/real 1789763644] req@ffff9018c8b9b480 x1876699504381056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763644.095462:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504381056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763644.095472:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763644.095476:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763644.095480:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504381056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763644.095504:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504381184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763644.095525:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763644.095537:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763644/real 1789763644] req@ffff9018c8b99f80 x1876699504381184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763644.095551:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504381184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763644.095561:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763644.095564:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763644.095568:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504381184:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763644.734260:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763645.758286:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763646.782366:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763647.806263:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763648.830271:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763649.214172:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763649.214188:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763649.214194:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763649.214210:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3575 00000100:00080000:0.0:1789763649.214225:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763649.214236:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763649.214282:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504381312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763649.214331:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763649.214336:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763649.214338:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763649.214343:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3575 00000100:00080000:0.0:1789763649.214363:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763649.214368:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763649.214378:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763649.214391:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504381440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763649.214398:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504381312:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763649.214404:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763649.214409:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763649.214426:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763649.214432:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3575 00000100:00080000:0.0:1789763649.214439:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763649.214443:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763649.214448:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763649.214463:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504381568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763649.214474:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763649.214478:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763649.214480:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763649.214483:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3575 00000100:00080000:0.0:1789763649.214488:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763649.214494:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763649.214499:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763649.214508:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763649.214522:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504381696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763649.214534:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763649.214538:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763649.214540:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763649.214543:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3575 00000100:00080000:0.0:1789763649.214546:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763649.214550:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763649.214555:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763649.214558:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763649/real 1789763649] req@ffff9018c8b98a80 x1876699504381312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763649.214571:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504381824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763649.214588:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504381312/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763649.214597:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763649.214611:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763649.214625:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763649.214629:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763649.214638:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3575 00000100:00080000:1.0:1789763649.214639:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763649.214651:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504381312:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763649.214653:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763649.214665:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763649.214673:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763649.214682:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504381440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763649.214698:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504381952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763649.214713:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763649.214729:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763649/real 1789763649] req@ffff9018c8b9a680 x1876699504381440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763649.214743:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504381440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763649.214753:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763649.214757:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763649.214760:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504381440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763649.214766:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504381568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763649.214797:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763649.214809:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763649/real 1789763649] req@ffff9018c8b98e00 x1876699504381568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763649.214821:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504381568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763649.214836:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763649.214839:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763649.214842:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504381568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763649.214848:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504381696:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763649.214864:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763649.214873:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763649/real 1789763649] req@ffff9018c8b9aa00 x1876699504381696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763649.214909:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504381696/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763649.214924:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763649.214929:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763649.214936:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504381696:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763649.214964:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504381824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763649.214999:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763649.215035:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763649/real 1789763649] req@ffff9018c8b9b800 x1876699504381824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763649.215048:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504381824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763649.215057:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763649.215060:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763649.215109:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504381824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763649.215130:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504381952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763649.215154:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763649.215182:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763649/real 1789763649] req@ffff9018c8b98000 x1876699504381952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763649.215195:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504381952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763649.215204:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763649.215207:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763649.215210:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504381952:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763649.854134:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763650.878288:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763651.902203:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763652.926246:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763653.950300:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763654.334187:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763654.334214:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763654.334219:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763654.334232:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3580 00000100:00080000:2.0:1789763654.334246:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763654.334255:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763654.334328:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504382080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763654.334370:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763654.334376:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763654.334379:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763654.334384:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3580 00000100:00080000:2.0:1789763654.334389:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763654.334394:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763654.334409:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763654.334426:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504382208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763654.334445:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763654.334450:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763654.334453:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763654.334463:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504382080:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763654.334464:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3580 00000100:00080000:2.0:1789763654.334475:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763654.334483:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763654.334490:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763654.334506:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504382336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763654.334520:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763654.334525:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763654.334528:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763654.334532:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3580 00000100:00080000:2.0:1789763654.334537:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763654.334542:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763654.334549:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763654.334568:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763654.334569:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504382464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763654.334588:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763654.334593:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763654.334597:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763654.334602:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3580 00000100:00080000:2.0:1789763654.334606:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763654.334611:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763654.334618:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763654.334632:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504382592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763654.334645:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763654.334647:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763654/real 1789763654] req@ffff9018d0f26d80 x1876699504382080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763654.334654:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763654.334661:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763654.334677:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3580 00000100:00080000:2.0:1789763654.334686:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763654.334692:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763654.334698:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763654.334704:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504382080/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763654.334721:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504382720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763654.334727:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763654.334743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763654.334763:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504382080:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763654.334811:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504382208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763654.334857:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763654.334887:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763654/real 1789763654] req@ffff9018d0f25f80 x1876699504382208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763654.334905:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504382208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763654.334916:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763654.334920:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763654.334925:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504382208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763654.334933:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504382336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763654.334961:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763654.334973:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763654/real 1789763654] req@ffff9018d0f27100 x1876699504382336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763654.334987:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504382336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763654.334997:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763654.335001:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763654.335004:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504382336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763654.335011:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504382464:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763654.335030:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763654.335094:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763654/real 1789763654] req@ffff9018d0f25180 x1876699504382464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763654.335110:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504382464/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763654.335120:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763654.335124:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763654.335128:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504382464:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763654.335135:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504382592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763654.335159:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763654.335170:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763654/real 1789763654] req@ffff9018d0f24380 x1876699504382592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763654.335184:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504382592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763654.335194:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763654.335198:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763654.335201:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504382592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763654.335207:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504382720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763654.335226:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763654.335237:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763654/real 1789763654] req@ffff9018d0f26300 x1876699504382720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763654.335251:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504382720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763654.335260:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763654.335263:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763654.335296:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504382720:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763654.974292:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763655.998282:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763657.022272:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763658.046317:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763659.070352:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763659.454224:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763659.454257:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763659.454263:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763659.454277:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3586 00000100:00080000:0.0:1789763659.454291:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763659.454300:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763659.454356:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504382848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763659.454388:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763659.454391:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763659.454394:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763659.454398:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3586 00000100:00080000:0.0:1789763659.454401:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763659.454405:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763659.454417:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763659.454429:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504382976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763659.454444:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763659.454451:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789763659.454451:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504382848:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763659.454457:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763659.454467:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3586 00000100:00080000:0.0:1789763659.454472:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763659.454478:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763659.454485:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763659.454503:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504383104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763659.454517:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763659.454522:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763659.454524:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763659.454529:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3586 00000100:00080000:0.0:1789763659.454543:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763659.454547:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763659.454553:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763659.454560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763659.454570:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504383232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763659.454582:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763659.454586:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763659.454587:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763659.454591:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3586 00000100:00080000:0.0:1789763659.454593:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763659.454596:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763659.454600:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763659.454613:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504383360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763659.454622:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763659/real 1789763659] req@ffff9018c8b99880 x1876699504382848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763659.454625:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763659.454635:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763659.454639:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763659.454649:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3586 00000100:00080000:0.0:1789763659.454653:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763659.454656:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763659.454661:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504382848/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763659.454662:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763659.454678:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504383488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763659.454680:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763659.454691:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763659.454699:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504382848:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763659.454732:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504382976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763659.454754:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763659.454766:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763659/real 1789763659] req@ffff9018c8b99500 x1876699504382976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763659.454777:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504382976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763659.454784:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763659.454788:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763659.454791:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504382976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763659.454795:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504383104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763659.454810:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763659.454818:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763659/real 1789763659] req@ffff9018c8b98380 x1876699504383104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763659.454828:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504383104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763659.454835:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763659.454837:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763659.454839:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504383104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763659.454844:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504383232:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763659.454858:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763659.454866:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763659/real 1789763659] req@ffff9018c8b9a300 x1876699504383232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763659.454876:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504383232/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763659.454883:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763659.454885:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763659.454887:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504383232:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763659.454891:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504383360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763659.454904:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763659.454911:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763659/real 1789763659] req@ffff9018c8b9ad80 x1876699504383360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763659.454921:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504383360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763659.454928:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763659.454931:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763659.454934:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504383360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763659.454938:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504383488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763659.454950:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763659.454957:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763659/real 1789763659] req@ffff9018c8b99c00 x1876699504383488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763659.454965:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504383488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763659.454971:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763659.454974:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763659.454977:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504383488:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763660.094301:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763661.118291:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763662.142270:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763663.166288:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763664.190322:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763664.574149:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763664.574162:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763664.574168:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763664.574181:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3591 00000100:00080000:0.0:1789763664.574218:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763664.574229:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763664.574291:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504383616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763664.574338:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763664.574343:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763664.574345:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763664.574351:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3591 00000100:00080000:0.0:1789763664.574355:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763664.574360:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763664.574373:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763664.574390:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504383744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763664.574405:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763664.574410:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763664.574412:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763664.574423:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3591 00000100:00100000:1.0:1789763664.574427:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504383616:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763664.574428:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763664.574440:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763664.574450:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763664.574466:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504383872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763664.574489:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763664.574494:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763664.574497:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763664.574502:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3591 00000100:00080000:0.0:1789763664.574509:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763664.574515:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763664.574521:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763664.574532:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763664.574549:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504384000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763664.574566:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763664.574570:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763664.574573:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763664.574577:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3591 00000100:00080000:0.0:1789763664.574582:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763664.574587:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763664.574594:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763664.574599:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763664/real 1789763664] req@ffff9018c8b99c00 x1876699504383616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763664.574614:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504384128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763664.574634:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763664.574641:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504383616/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763664.574642:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763664.574649:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763664.574664:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3591 00000100:00080000:0.0:1789763664.574675:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763664.574682:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763664.574683:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763664.574698:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763664.574701:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763664.574713:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504383616:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763664.574717:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504384256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763664.574733:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504383744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763664.574770:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763664.574787:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763664/real 1789763664] req@ffff9018c8b9ad80 x1876699504383744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763664.574803:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504383744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763664.574824:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763664.574828:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763664.574832:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504383744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763664.574861:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504383872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763664.574882:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763664.574894:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763664/real 1789763664] req@ffff9018c8b9a300 x1876699504383872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763664.574908:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504383872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763664.574917:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763664.574921:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763664.574940:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504383872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763664.574949:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504384000:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763664.574982:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763664.574996:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763664/real 1789763664] req@ffff9018c8b98380 x1876699504384000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763664.575010:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504384000/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763664.575021:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763664.575024:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763664.575028:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504384000:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763664.575034:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504384128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763664.575100:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763664.575115:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763664/real 1789763664] req@ffff9018c8b99500 x1876699504384128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763664.575130:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504384128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763664.575140:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763664.575144:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763664.575147:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504384128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763664.575154:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504384256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763664.575172:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763664.575183:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763664/real 1789763664] req@ffff9018c8b99880 x1876699504384256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763664.575230:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504384256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763664.575256:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763664.575263:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763664.575270:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504384256:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763665.214469:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763666.238300:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763667.262294:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763668.286333:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763669.310278:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763669.694143:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763669.694191:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763669.694196:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763669.694207:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3596 00000100:00080000:0.0:1789763669.694220:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763669.694227:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763669.694292:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504384384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763669.694337:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763669.694341:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763669.694346:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763669.694350:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3596 00000100:00080000:0.0:1789763669.694353:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763669.694357:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763669.694369:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763669.694381:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504384512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763669.694389:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504384384:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763669.694423:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763669.694426:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763669.694428:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763669.694432:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3596 00000100:00080000:0.0:1789763669.694435:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763669.694438:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763669.694443:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763669.694455:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504384640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763669.694462:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763669.694466:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763669.694468:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763669.694471:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3596 00000100:00080000:0.0:1789763669.694484:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763669.694487:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763669.694508:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763669.694520:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504384768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763669.694526:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763669.694537:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763669.694541:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763669.694543:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763669.694548:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3596 00000100:00080000:0.0:1789763669.694553:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763669.694556:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763669.694560:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763669.694569:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504384896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763669.694578:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763669.694582:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763669/real 1789763669] req@ffff9018c8b98000 x1876699504384384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763669.694583:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763669.694588:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763669.694597:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3596 00000100:00080000:0.0:1789763669.694603:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763669.694606:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763669.694610:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763669.694613:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504384384/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763669.694626:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504385024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763669.694627:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763669.694636:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763669.694644:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504384384:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763669.694675:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504384512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763669.694697:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763669.694711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763669/real 1789763669] req@ffff9018c8b9b800 x1876699504384512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763669.694722:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504384512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763669.694729:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763669.694732:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763669.694735:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504384512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763669.694740:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504384640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763669.694753:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763669.694760:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763669/real 1789763669] req@ffff9018c8b9aa00 x1876699504384640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763669.694769:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504384640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763669.694775:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763669.694777:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763669.694779:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504384640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763669.694784:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504384768:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763669.694796:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763669.694803:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763669/real 1789763669] req@ffff9018c8b98e00 x1876699504384768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763669.694812:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504384768/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763669.694818:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763669.694820:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763669.694822:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504384768:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763669.694826:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504384896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763669.694837:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763669.694844:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763669/real 1789763669] req@ffff9018c8b9a680 x1876699504384896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763669.694853:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504384896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763669.694859:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763669.694861:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763669.694863:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504384896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763669.694867:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504385024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763669.694880:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763669.694891:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763669/real 1789763669] req@ffff9018c8b98a80 x1876699504385024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763669.694905:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504385024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763669.694914:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763669.694918:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763669.694922:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504385024:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763670.334245:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763671.358315:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763672.382256:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763673.406291:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763674.430311:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763674.814180:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763674.814197:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763674.814212:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763674.814224:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3601 00000100:00080000:2.0:1789763674.814237:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763674.814249:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763674.814298:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504385152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763674.814331:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763674.814335:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763674.814336:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763674.814340:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3601 00000100:00080000:2.0:1789763674.814343:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763674.814345:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763674.814356:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763674.814368:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504385280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763674.814377:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763674.814380:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763674.814381:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763674.814388:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504385152:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763674.814388:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3601 00000100:00080000:2.0:1789763674.814394:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763674.814399:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763674.814403:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763674.814412:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504385408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763674.814419:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763674.814422:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763674.814423:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763674.814426:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3601 00000100:00080000:2.0:1789763674.814430:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763674.814433:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763674.814438:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763674.814447:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504385536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763674.814455:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763674.814458:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763674.814459:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763674.814462:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3601 00000100:00080000:2.0:1789763674.814465:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763674.814468:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763674.814471:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763674.814478:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763674.814483:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504385664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763674.814493:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763674.814497:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763674.814499:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763674.814502:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3601 00000100:00080000:2.0:1789763674.814505:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763674.814509:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763674.814512:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763674.814519:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504385792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763674.814542:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763674/real 1789763674] req@ffff9018d0f24000 x1876699504385152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763674.814572:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504385152/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763674.814589:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763674.814593:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763674.814596:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504385152:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763674.814624:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504385280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763674.814642:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763674.814652:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763674/real 1789763674] req@ffff9018d0f26680 x1876699504385280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763674.814661:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504385280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763674.814668:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763674.814671:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763674.814673:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504385280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763674.814677:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504385408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763674.814689:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763674.814695:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763674/real 1789763674] req@ffff9018d0f26a00 x1876699504385408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763674.814703:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504385408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763674.814709:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763674.814712:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763674.814714:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504385408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763674.814719:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504385536:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763674.814732:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763674.814739:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763674/real 1789763674] req@ffff9018d0f25c00 x1876699504385536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763674.814748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504385536/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763674.814754:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763674.814756:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763674.814758:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504385536:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763674.814761:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504385664:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763674.814772:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763674.814779:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763674/real 1789763674] req@ffff9018d0f27800 x1876699504385664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763674.814787:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504385664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763674.814793:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763674.814795:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763674.814797:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504385664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763674.814801:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504385792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763674.814812:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763674.814818:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763674/real 1789763674] req@ffff9018d0f24a80 x1876699504385792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763674.814827:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504385792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763674.814833:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763674.814835:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763674.814837:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504385792:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763675.454345:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763676.478279:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763677.502282:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763678.526341:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763679.550261:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763679.934259:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763679.934269:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763679.934274:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763679.934286:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3606 00000100:00080000:0.0:1789763679.934297:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763679.934310:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763679.934353:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504385920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763679.934393:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763679.934397:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763679.934399:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763679.934403:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3606 00000100:00080000:0.0:1789763679.934407:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763679.934410:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763679.934422:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763679.934439:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504386048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763679.934452:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763679.934456:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763679.934459:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763679.934466:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3606 00000100:00100000:2.0:1789763679.934476:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504385920:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763679.934477:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763679.934490:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763679.934502:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763679.934523:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504386176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763679.934534:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763679.934538:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763679.934540:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763679.934543:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3606 00000100:00080000:0.0:1789763679.934549:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763679.934553:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763679.934558:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763679.934569:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504386304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763679.934580:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763679.934584:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763679.934586:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763679.934589:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3606 00000100:00080000:0.0:1789763679.934594:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763679.934599:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763679.934603:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763679.934615:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504386432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763679.934629:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763679.934637:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763679.934642:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763679.934644:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763679.934653:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3606 00000100:00080000:0.0:1789763679.934660:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763679.934668:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763679.934676:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763679.934691:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504386560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763679.934736:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763679/real 1789763679] req@ffff9018c8b99f80 x1876699504385920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763679.934786:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504385920/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763679.934811:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763679.934820:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763679.934829:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504385920:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763679.934879:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504386048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763679.934937:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763679.934972:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763679/real 1789763679] req@ffff9018c8b9b480 x1876699504386048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763679.934995:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504386048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763679.935008:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763679.935014:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763679.935025:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504386048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763679.935114:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504386176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763679.935183:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763679.935212:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763679/real 1789763679] req@ffff9018c8b9b100 x1876699504386176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763679.935228:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504386176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763679.935238:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763679.935243:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763679.935247:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504386176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763679.935256:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504386304:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763679.935280:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763679.935293:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763679/real 1789763679] req@ffff9018c8b9bb80 x1876699504386304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763679.935308:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504386304/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763679.935318:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763679.935322:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763679.935326:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504386304:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763679.935333:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504386432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763679.935354:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763679.935365:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763679/real 1789763679] req@ffff9018c8b98700 x1876699504386432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763679.935378:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504386432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763679.935390:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763679.935394:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763679.935398:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504386432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763679.935404:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504386560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763679.935424:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763679.935435:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763679/real 1789763679] req@ffff9018c8b98a80 x1876699504386560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763679.935448:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504386560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763679.935457:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763679.935461:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763679.935465:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504386560:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763680.574273:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763681.598291:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763682.622304:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763683.646176:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763684.670295:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763685.054177:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763685.054192:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763685.054199:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763685.054213:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3611 00000100:00080000:0.0:1789763685.054228:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763685.054239:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763685.054292:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504386688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763685.054331:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763685.054336:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763685.054339:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763685.054344:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3611 00000100:00080000:0.0:1789763685.054348:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763685.054354:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763685.054368:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763685.054387:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504386816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763685.054411:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763685.054413:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504386688:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763685.054437:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763685.054444:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763685.054451:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3611 00000100:00080000:0.0:1789763685.054455:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763685.054459:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763685.054466:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763685.054489:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504386944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763685.054503:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763685.054507:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763685.054510:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763685.054514:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3611 00000100:00080000:0.0:1789763685.054523:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763685.054529:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763685.054536:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763685.054550:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504387072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763685.054561:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763685.054566:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763685.054569:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763685.054574:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3611 00000100:00080000:0.0:1789763685.054581:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763685.054591:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763685.054600:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763685.054609:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763685.054640:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504387200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763685.054664:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763685.054674:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789763685.054676:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763685/real 1789763685] req@ffff9018c8b9a680 x1876699504386688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763685.054682:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763685.054710:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3611 00000100:00080000:0.0:1789763685.054730:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763685.054740:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763685.054747:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763685.054748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504386688/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763685.054773:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504387328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763685.054777:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763685.054802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763685.054822:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504386688:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763685.054893:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504386816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763685.054953:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763685.054988:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763685/real 1789763685] req@ffff9018c8b98e00 x1876699504386816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763685.055008:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504386816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763685.055021:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763685.055026:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763685.055105:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504386816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763685.055119:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504386944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763685.055152:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763685.055168:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763685/real 1789763685] req@ffff9018c8b9aa00 x1876699504386944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763685.055183:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504386944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763685.055194:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763685.055199:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763685.055203:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504386944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763685.055211:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504387072:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763685.055232:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763685.055245:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763685/real 1789763685] req@ffff9018c8b9b800 x1876699504387072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763685.055260:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504387072/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763685.055270:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763685.055274:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763685.055278:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504387072:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763685.055285:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504387200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763685.055304:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763685.055316:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763685/real 1789763685] req@ffff9018c8b98000 x1876699504387200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763685.055331:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504387200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763685.055340:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763685.055344:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763685.055347:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504387200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763685.055354:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504387328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763685.055374:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763685.055384:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763685/real 1789763685] req@ffff9018c8b99880 x1876699504387328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763685.055397:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504387328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763685.055408:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763685.055412:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763685.055416:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504387328:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763685.694236:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763686.718230:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763687.742272:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763688.766288:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763689.790251:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763690.174286:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763690.174301:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763690.174307:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763690.174323:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3616 00000100:00080000:3.0:1789763690.174342:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763690.174353:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763690.174418:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504387456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763690.174463:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763690.174469:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763690.174471:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763690.174477:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3616 00000100:00080000:3.0:1789763690.174481:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763690.174487:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763690.174503:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763690.174520:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504387584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763690.174536:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763690.174540:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763690.174543:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763690.174553:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504387456:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763690.174554:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3616 00000100:00080000:3.0:1789763690.174563:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763690.174573:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763690.174579:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763690.174594:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504387712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763690.174607:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763690.174611:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763690.174615:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763690.174619:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3616 00000100:00080000:3.0:1789763690.174623:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763690.174627:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763690.174634:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763690.174649:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504387840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763690.174661:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763690.174667:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763690.174670:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763690.174675:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3616 00000100:00080000:3.0:1789763690.174680:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763690.174685:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763690.174690:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763690.174704:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504387968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763690.174717:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763690.174724:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763690.174728:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763690.174733:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3616 00000100:00080000:3.0:1789763690.174738:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763690.174743:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763690.174750:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504387584:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789763690.174754:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763690.174783:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504388096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763690.174812:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504387712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763690.174843:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504387840:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763690.174877:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504387968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763690.174912:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504388096:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763690.814225:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763691.838258:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763692.862233:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763693.246445:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:0.0:1789763693.246517:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763693.246544:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763693.246612:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763693.246629:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763693.246639:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763693.246650:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763693.246666:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:2.0:1789763693.246740:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763690/real 1789763693] req@ffff9018d070d880 x1876699504388096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763693.246791:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504388096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763693.246814:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763693.246823:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763693.246832:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504388096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763693.246847:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763690/real 1789763693] req@ffff9018d070fb80 x1876699504387584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763693.246886:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504387584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763693.246904:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763693.246912:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763693.246919:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504387584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763693.246933:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763690/real 1789763693] req@ffff9018d070f480 x1876699504387712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763693.246957:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504387712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763693.246973:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763693.246980:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763693.247010:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504387712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763693.247025:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763690/real 1789763693] req@ffff9018d070ce00 x1876699504387840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763693.247109:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504387840/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763693.247125:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763693.247132:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763693.247139:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504387840:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763693.247152:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763690/real 1789763693] req@ffff9018d070ed80 x1876699504387968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763693.247179:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504387968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763693.247194:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763693.247202:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763693.247208:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504387968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763693.247223:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763690/real 1789763693] req@ffff9018d070df80 x1876699504387456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763693.247248:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504387456/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763693.247263:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763693.247271:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763693.247277:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504387456:192.168.203.158@tcp:250:kworker.0 00010000:00010000:3.0:1789763693.886261:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763694.910345:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763695.294208:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763695.294223:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763695.294229:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763695.294247:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3621 00000100:00080000:3.0:1789763695.294265:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763695.294277:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763695.294339:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504388224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763695.294382:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763695.294387:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763695.294390:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763695.294396:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3621 00000100:00080000:3.0:1789763695.294400:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763695.294404:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763695.294419:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763695.294436:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504388352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763695.294450:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763695.294454:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763695.294457:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763695.294461:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3621 00000100:00080000:3.0:1789763695.294465:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763695.294469:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763695.294476:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763695.294511:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504388480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763695.294513:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504388224:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763695.294543:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763695.294548:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763695.294550:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763695.294553:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3621 00000100:00080000:3.0:1789763695.294557:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763695.294561:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763695.294576:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763695.294598:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504388608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763695.294616:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763695.294623:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763695.294627:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763695.294633:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3621 00000100:00080000:3.0:1789763695.294653:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763695.294662:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789763695.294669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789763695.294678:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763695.294705:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504388736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763695.294724:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763695.294730:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763695.294734:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763695.294741:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3621 00000100:00080000:3.0:1789763695.294750:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763695.294759:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763695/real 1789763695] req@ffff9018d070ca80 x1876699504388224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763695.294762:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763695.294777:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763695.294806:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504388864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763695.294818:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504388224/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763695.294849:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763695.294857:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763695.294868:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504388224:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763695.294894:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504388352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763695.294941:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763695.294995:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763695/real 1789763695] req@ffff9018d070d180 x1876699504388352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763695.295022:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504388352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763695.295097:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763695.295105:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763695.295112:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504388352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763695.295162:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504388480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763695.295205:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763695.295228:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763695/real 1789763695] req@ffff9018d070e680 x1876699504388480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763695.295249:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504388480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763695.295262:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763695.295268:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763695.295273:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504388480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763695.295283:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504388608:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763695.295318:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763695.295339:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763695/real 1789763695] req@ffff9018d070d500 x1876699504388608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763695.295361:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504388608/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763695.295375:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763695.295381:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763695.295387:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504388608:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763695.295397:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504388736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763695.295435:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763695.295470:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763695/real 1789763695] req@ffff9018d070dc00 x1876699504388736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763695.295492:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504388736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763695.295507:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763695.295513:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763695.295520:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504388736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763695.295534:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504388864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763695.295578:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763695.295600:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763695/real 1789763695] req@ffff9018d070c000 x1876699504388864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763695.295622:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504388864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763695.295649:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763695.295655:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763695.295661:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504388864:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763695.934187:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763696.958268:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763697.982265:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763699.006269:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763700.030224:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763700.414197:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763700.414213:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763700.414221:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763700.414238:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3626 00000100:00080000:0.0:1789763700.414255:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763700.414267:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763700.414328:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504388992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763700.414371:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763700.414377:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763700.414380:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763700.414386:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3626 00000100:00080000:0.0:1789763700.414390:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763700.414395:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763700.414412:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763700.414435:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504389120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763700.414456:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763700.414464:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763700.414469:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763700.414478:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3626 00000100:00080000:0.0:1789763700.414487:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763700.414501:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763700.414501:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504388992:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789763700.414530:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763700.414555:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504389248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763700.414580:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763700.414585:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763700.414587:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763700.414592:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3626 00000100:00080000:0.0:1789763700.414600:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763700.414605:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763700.414613:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789763700.414622:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763700.414638:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504389376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763700.414654:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763700.414660:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763700.414662:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763700.414666:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3626 00000100:00080000:0.0:1789763700.414670:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763700.414676:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763700.414683:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763700.414694:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763700/real 1789763700] req@ffff9018c8b99500 x1876699504388992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763700.414701:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504389504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763700.414746:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763700.414752:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763700.414755:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763700.414759:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504388992/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763700.414767:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3626 00000100:00080000:0.0:1789763700.414778:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763700.414790:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789763700.414797:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789763700.414798:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763700.414812:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763700.414823:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504389632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763700.414825:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504388992:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763700.414851:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504389120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763700.414887:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763700.414906:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763700/real 1789763700] req@ffff9018c8b98380 x1876699504389120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763700.414921:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504389120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763700.414964:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763700.414968:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763700.414973:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504389120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763700.415012:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504389248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763700.415082:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763700.415099:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763700/real 1789763700] req@ffff9018c8b9a300 x1876699504389248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763700.415114:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504389248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763700.415124:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763700.415129:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763700.415133:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504389248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763700.415140:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504389376:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763700.415164:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763700.415176:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763700/real 1789763700] req@ffff9018c8b9ad80 x1876699504389376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763700.415190:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504389376/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763700.415201:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763700.415204:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763700.415207:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504389376:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763700.415213:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504389504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763700.415233:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763700.415244:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763700/real 1789763700] req@ffff9018c8b99c00 x1876699504389504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763700.415259:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504389504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763700.415268:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763700.415271:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763700.415275:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504389504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763700.415281:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504389632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763700.415300:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763700.415310:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763700/real 1789763700] req@ffff9018c8b99880 x1876699504389632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763700.415323:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504389632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763700.415332:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763700.415335:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763700.415339:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504389632:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763701.054252:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763702.078355:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763703.102260:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763704.126277:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763705.150276:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763705.534179:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763705.534190:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763705.534194:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763705.534203:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3632 00000100:00080000:3.0:1789763705.534215:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763705.534223:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763705.534264:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504389760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763705.534295:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763705.534299:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763705.534300:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763705.534305:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3632 00000100:00080000:3.0:1789763705.534308:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763705.534311:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763705.534322:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763705.534333:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504389888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763705.534343:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763705.534346:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763705.534347:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763705.534350:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3632 00000100:00080000:3.0:1789763705.534353:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763705.534355:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763705.534359:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763705.534367:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504390016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763705.534376:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763705.534379:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763705.534381:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763705.534386:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3632 00000100:00080000:3.0:1789763705.534389:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763705.534393:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763705.534398:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763705.534409:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504390144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763705.534420:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763705.534423:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763705.534426:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763705.534437:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504389760:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763705.534455:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3632 00000100:00080000:3.0:1789763705.534462:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763705.534468:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763705.534475:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763705.534490:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504390272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763705.534502:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763705.534507:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763705.534509:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763705.534514:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3632 00000100:00080000:3.0:1789763705.534518:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763705.534523:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763705.534531:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763705.534543:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504390400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763705.534570:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763705.534630:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763705/real 1789763705] req@ffff9018d070e300 x1876699504389760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763705.534649:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504389760/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763705.534668:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763705.534672:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763705.534676:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504389760:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763705.534702:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504389888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763705.534732:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763705.534747:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763705/real 1789763705] req@ffff9018d070f100 x1876699504389888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763705.534763:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504389888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763705.534774:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763705.534778:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763705.534782:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504389888:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763705.534789:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504390016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763705.534809:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763705.534820:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763705/real 1789763705] req@ffff9018d070f800 x1876699504390016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763705.534834:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504390016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763705.534844:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763705.534847:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763705.534851:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504390016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763705.534921:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504390144:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763705.534971:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763705.534981:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763705/real 1789763705] req@ffff9018d070c000 x1876699504390144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763705.534990:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504390144/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763705.534998:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763705.535001:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763705.535004:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504390144:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763705.535008:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504390272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763705.535020:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763705.535027:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763705/real 1789763705] req@ffff9018d070dc00 x1876699504390272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763705.535036:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504390272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763705.535043:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763705.535045:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763705.535047:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504390272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763705.535094:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504390400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763705.535110:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763705.535119:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763705/real 1789763705] req@ffff9018d070d500 x1876699504390400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763705.535129:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504390400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763705.535136:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763705.535138:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763705.535141:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504390400:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763706.174218:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763707.198247:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763708.222359:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763709.246328:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763710.270114:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763710.654197:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763710.654205:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763710.654208:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763710.654219:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3637 00000100:00080000:1.0:1789763710.654232:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763710.654240:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763710.654269:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504390528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763710.654290:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763710.654293:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763710.654295:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763710.654301:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3637 00000100:00080000:1.0:1789763710.654304:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763710.654319:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763710.654329:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763710.654341:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504390528:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763710.654344:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504390656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763710.654365:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763710.654368:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763710.654371:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763710.654379:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3637 00000100:00080000:1.0:1789763710.654382:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763710.654386:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763710.654390:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763710.654400:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504390784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763710.654410:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763710.654411:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763710.654416:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763710.654417:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763710.654423:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3637 00000100:00080000:1.0:1789763710.654427:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763710.654431:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763710.654434:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763710.654444:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504390912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763710.654451:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763710.654454:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763710/real 1789763710] req@ffff9018c66a5180 x1876699504390528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763710.654455:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763710.654457:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763710.654462:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3637 00000100:00080000:1.0:1789763710.654466:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763710.654472:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763710.654479:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763710.654487:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504391040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763710.654492:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504390528/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763710.654496:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763710.654503:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763710.654506:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763710.654514:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3637 00000100:00080000:1.0:1789763710.654517:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763710.654519:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:2.0:1789763710.654519:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1789763710.654528:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763710.654531:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763710.654537:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504391168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763710.654540:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504390528:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763710.654576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504390656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763710.654598:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763710.654608:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763710/real 1789763710] req@ffff9018c66a6300 x1876699504390656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763710.654617:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504390656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763710.654651:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763710.654656:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763710.654660:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504390656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763710.654669:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504390784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763710.654697:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763710.654708:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763710/real 1789763710] req@ffff9018c66a4700 x1876699504390784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763710.654722:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504390784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763710.654732:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763710.654736:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763710.654739:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504390784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763710.654746:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504390912:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763710.654765:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763710.654776:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763710/real 1789763710] req@ffff9018c66a6680 x1876699504390912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763710.654790:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504390912/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763710.654800:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763710.654803:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763710.654808:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504390912:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763710.654816:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504391040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763710.654834:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763710.654845:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763710/real 1789763710] req@ffff9018c66a5500 x1876699504391040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763710.654893:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504391040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763710.654923:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763710.654925:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763710.654928:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504391040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763710.654932:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504391168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763710.654945:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763710.654952:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763710/real 1789763710] req@ffff9018c66a4e00 x1876699504391168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763710.654971:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504391168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763710.654975:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763710.654977:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763710.654979:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504391168:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763711.294154:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763712.318291:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763713.342232:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763714.366218:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763715.390237:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763715.774171:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763715.774185:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763715.774191:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763715.774206:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3642 00000100:00080000:1.0:1789763715.774221:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763715.774231:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763715.774278:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504391296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763715.774332:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763715.774340:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763715.774343:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763715.774348:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3642 00000100:00080000:1.0:1789763715.774353:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763715.774359:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763715.774373:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763715.774398:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504391424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763715.774415:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763715.774420:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763715.774423:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763715.774433:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3642 00000100:00100000:0.0:1789763715.774439:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504391296:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763715.774440:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763715.774454:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763715.774463:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763715.774481:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504391552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763715.774498:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763715.774505:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763715.774508:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763715.774514:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3642 00000100:00080000:1.0:1789763715.774535:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763715.774541:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763715.774551:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789763715.774555:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763715.774573:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504391680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763715.774592:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763715.774597:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763715.774599:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763715.774606:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3642 00000100:00080000:1.0:1789763715.774612:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763715.774620:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763715.774628:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763715.774633:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763715/real 1789763715] req@ffff9018c66a5880 x1876699504391296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763715.774651:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504391808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763715.774682:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763715.774686:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763715.774689:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763715.774699:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504391296/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763715.774700:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3642 00000100:00080000:1.0:1789763715.774713:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763715.774726:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763715.774738:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789763715.774747:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789763715.774756:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504391936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763715.774766:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763715.774786:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504391296:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763715.774806:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504391424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763715.774892:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763715.774931:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763715/real 1789763715] req@ffff9018c66a7480 x1876699504391424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763715.774972:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504391424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763715.774998:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763715.775007:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763715.775016:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504391424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763715.775159:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504391552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763715.775212:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763715.775249:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763715/real 1789763715] req@ffff9018c66a4a80 x1876699504391552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763715.775273:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504391552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763715.775289:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763715.775295:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763715.775302:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504391552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763715.775314:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504391680:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763715.775353:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763715.775376:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763715/real 1789763715] req@ffff9018c66a7100 x1876699504391680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763715.775398:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504391680/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763715.775425:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763715.775431:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763715.775438:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504391680:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763715.775461:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504391808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763715.775500:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763715.775523:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763715/real 1789763715] req@ffff9018c66a4380 x1876699504391808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763715.775547:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504391808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763715.775563:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763715.775569:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763715.775575:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504391808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763715.775585:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504391936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763715.775621:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763715.775645:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763715/real 1789763715] req@ffff9018c66a6a00 x1876699504391936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763715.775668:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504391936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763715.775684:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763715.775691:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763715.775698:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504391936:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763716.414266:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763717.438294:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763718.462315:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763719.486273:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763720.510264:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763720.894166:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763720.894180:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763720.894186:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763720.894198:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3647 00000100:00080000:1.0:1789763720.894212:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763720.894221:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763720.894270:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504392064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763720.894306:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763720.894311:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763720.894314:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763720.894319:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3647 00000100:00080000:1.0:1789763720.894323:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763720.894328:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763720.894342:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763720.894357:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504392192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763720.894373:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763720.894380:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504392064:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763720.894381:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763720.894390:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763720.894404:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3647 00000100:00080000:1.0:1789763720.894408:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763720.894413:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763720.894420:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763720.894433:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504392320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763720.894455:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763720.894460:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763720.894476:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763720.894482:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3647 00000100:00080000:1.0:1789763720.894487:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763720.894497:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763720.894497:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763720.894514:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763720.894541:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504392448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763720.894558:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763720/real 1789763720] req@ffff9018c66a6a00 x1876699504392064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763720.894566:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763720.894588:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763720.894595:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763720.894603:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504392064/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763720.894606:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3647 00000100:00080000:1.0:1789763720.894622:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763720.894628:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763720.894634:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789763720.894642:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:1.0:1789763720.894650:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763720.894651:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504392064:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763720.894683:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504392576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763720.894698:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504392192:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789763720.894705:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763720.894715:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763720.894717:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763720.894724:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3647 00000100:00080000:1.0:1789763720.894730:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763720.894736:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763720.894743:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763720.894766:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504392704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763720.894767:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763720.894834:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763720/real 1789763720] req@ffff9018c66a4380 x1876699504392192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763720.894857:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504392192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763720.894869:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763720.894873:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763720.894878:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504392192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763720.894886:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504392320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763720.894917:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763720.894931:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763720/real 1789763720] req@ffff9018c66a7100 x1876699504392320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763720.894946:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504392320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763720.894956:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763720.894959:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763720.894963:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504392320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763720.894970:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504392448:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763720.894990:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763720.895001:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763720/real 1789763720] req@ffff9018c66a4a80 x1876699504392448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763720.895015:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504392448/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763720.895025:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763720.895029:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763720.895074:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504392448:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763720.895100:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504392576:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763720.895126:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763720.895140:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763720/real 1789763720] req@ffff9018c66a7480 x1876699504392576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763720.895177:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504392576/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763720.895194:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763720.895200:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763720.895207:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504392576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763720.895219:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504392704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763720.895257:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763720.895275:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763720/real 1789763720] req@ffff9018c66a5880 x1876699504392704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763720.895291:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504392704/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763720.895301:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763720.895305:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763720.895309:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504392704:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763721.534264:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763722.558206:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763723.582269:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763724.606295:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763725.630417:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763726.014162:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763726.014173:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763726.014178:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763726.014188:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3652 00000100:00080000:2.0:1789763726.014200:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763726.014208:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763726.014249:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504392832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763726.014279:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763726.014283:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763726.014285:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763726.014289:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3652 00000100:00080000:2.0:1789763726.014292:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763726.014296:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763726.014306:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763726.014318:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504392960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763726.014329:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763726.014332:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763726.014335:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763726.014338:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3652 00000100:00080000:2.0:1789763726.014341:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763726.014344:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763726.014349:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763726.014354:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504392832:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763726.014360:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504393088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763726.014440:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763726.014443:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763726.014445:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763726.014449:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3652 00000800:00000100:0.0:1789763726.014456:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763726.014457:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763726.014467:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763726.014474:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763726.014488:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504393216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763726.014499:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763726.014503:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763726.014505:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763726.014509:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3652 00000100:00080000:2.0:1789763726.014513:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763726.014517:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763726/real 1789763726] req@ffff9018d0f25500 x1876699504392832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763726.014523:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763726.014532:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763726.014555:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504393344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763726.014557:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504392832/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763726.014591:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763726.014595:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763726.014596:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763726.014604:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763726.014608:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763726.014615:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3652 00000100:00100000:0.0:1789763726.014620:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504392832:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763726.014623:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763726.014635:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763726.014640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504392960:192.168.203.158@tcp:38:kworker.0 00000008:00100000:2.0:1789763726.014642:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763726.014668:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504393472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763726.014688:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763726.014716:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763726/real 1789763726] req@ffff9018d0f27480 x1876699504392960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763726.014739:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504392960/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763726.014768:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763726.014772:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763726.014776:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504392960:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763726.014802:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504393088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763726.014826:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763726.014838:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763726/real 1789763726] req@ffff9018d0f24e00 x1876699504393088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763726.014850:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504393088/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763726.014857:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763726.014861:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763726.014864:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504393088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763726.014870:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504393216:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763726.014885:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763726.014894:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763726/real 1789763726] req@ffff9018d0f24700 x1876699504393216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763726.014905:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504393216/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763726.014913:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763726.014915:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763726.014918:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504393216:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763726.014923:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504393344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763726.014937:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763726.014945:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763726/real 1789763726] req@ffff9018d0f24a80 x1876699504393344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763726.014962:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504393344/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763726.014974:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763726.014979:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763726.014984:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504393344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763726.014993:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504393472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763726.015024:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763726.015071:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763726/real 1789763726] req@ffff9018d0f27800 x1876699504393472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763726.015085:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504393472/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763726.015093:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763726.015096:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763726.015099:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504393472:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763726.654280:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763727.678315:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763728.702280:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763729.726184:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763730.750233:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763731.134287:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763731.134300:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763731.134306:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763731.134318:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3657 00000100:00080000:3.0:1789763731.134332:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763731.134341:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763731.134397:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504393600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763731.134448:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763731.134456:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763731.134459:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763731.134467:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3657 00000100:00080000:3.0:1789763731.134473:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763731.134481:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763731.134495:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763731.134518:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504393600:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763731.134521:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504393728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763731.134546:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763731.134551:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763731.134555:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763731.134560:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3657 00000100:00080000:3.0:1789763731.134564:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763731.134570:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763731.134576:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763731.134592:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504393856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763731.134604:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763731.134612:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1789763731.134614:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763731.134616:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763731.134626:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3657 00000100:00080000:3.0:1789763731.134634:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763731.134640:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763731.134647:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763731.134663:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504393984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763731.134677:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763731/real 1789763731] req@ffff9018d070e680 x1876699504393600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763731.134681:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763731.134696:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763731.134726:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763731.134741:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3657 00000100:00080000:3.0:1789763731.134748:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763731.134760:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763731.134762:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504393600/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789763731.134769:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789763731.134791:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789763731.134800:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504394112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763731.134802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763731.134818:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504393600:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763731.134835:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763731.134840:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763731.134842:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763731.134849:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3657 00000100:00080000:3.0:1789763731.134852:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763731.134858:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763731.134864:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763731.134867:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504393728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763731.134883:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504394240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763731.134916:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763731.134938:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763731/real 1789763731] req@ffff9018d070d180 x1876699504393728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763731.134956:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504393728/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763731.134968:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763731.134973:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763731.134989:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504393728:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763731.134997:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504393856:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763731.135021:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763731.135067:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763731/real 1789763731] req@ffff9018d070ca80 x1876699504393856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763731.135085:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504393856/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763731.135095:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763731.135099:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763731.135103:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504393856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763731.135110:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504393984:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763731.135134:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763731.135147:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763731/real 1789763731] req@ffff9018d070df80 x1876699504393984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763731.135161:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504393984/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763731.135171:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763731.135174:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763731.135179:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504393984:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763731.135186:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504394112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763731.135205:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763731.135215:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763731/real 1789763731] req@ffff9018d070ed80 x1876699504394112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763731.135238:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504394112/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763731.135253:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763731.135259:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763731.135266:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504394112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763731.135300:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504394240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763731.135341:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763731.135361:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763731/real 1789763731] req@ffff9018d070ce00 x1876699504394240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763731.135383:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504394240/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763731.135398:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763731.135404:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763731.135411:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504394240:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763731.774244:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763732.798218:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763733.822310:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763734.846254:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763735.870266:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763736.254131:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763736.254146:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763736.254153:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763736.254168:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3662 00000100:00080000:3.0:1789763736.254185:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763736.254195:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763736.254262:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504394368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763736.254311:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763736.254317:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763736.254321:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763736.254327:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3662 00000100:00080000:3.0:1789763736.254334:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763736.254341:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763736.254356:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763736.254377:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504394368:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763736.254380:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504394496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763736.254411:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763736.254416:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763736.254418:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763736.254423:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3662 00000100:00080000:3.0:1789763736.254428:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763736.254432:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763736.254438:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763736.254454:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504394624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763736.254477:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763736.254482:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763736.254485:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763736.254489:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3662 00000100:00080000:3.0:1789763736.254493:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763736.254498:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763736.254504:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763736.254518:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504394752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763736.254531:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763736.254559:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763736.254561:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763736.254573:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763736.254577:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3662 00000100:00080000:3.0:1789763736.254586:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763736.254592:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763736.254600:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763736.254616:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504394880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763736.254630:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763736.254636:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763736.254638:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763736.254644:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3662 00000100:00100000:0.0:1789763736.254652:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763736/real 1789763736] req@ffff9018d070f480 x1876699504394368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763736.254654:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763736.254693:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763736.254731:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763736.254751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504394368/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763736.254754:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504395008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763736.254821:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763736.254832:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763736.254844:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504394368:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763736.254892:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504394496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763736.254958:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763736.254978:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763736/real 1789763736] req@ffff9018d070fb80 x1876699504394496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763736.254996:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504394496/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763736.255008:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763736.255013:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763736.255017:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504394496:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763736.255025:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504394624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763736.255048:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763736.255062:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763736/real 1789763736] req@ffff9018d070d880 x1876699504394624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763736.255170:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504394624/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763736.255187:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763736.255194:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763736.255200:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504394624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763736.255213:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504394752:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763736.255256:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763736.255277:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763736/real 1789763736] req@ffff9018d070ce00 x1876699504394752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763736.255312:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504394752/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763736.255323:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763736.255327:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763736.255332:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504394752:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763736.255340:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504394880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763736.255368:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763736.255381:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763736/real 1789763736] req@ffff9018d070ed80 x1876699504394880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763736.255419:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504394880/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763736.255450:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763736.255458:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763736.255467:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504394880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763736.255482:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504395008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763736.255539:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763736.255556:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763736/real 1789763736] req@ffff9018d070df80 x1876699504395008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763736.255571:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504395008/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763736.255582:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763736.255587:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763736.255592:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504395008:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763736.894242:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763737.918360:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763738.942264:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763739.966228:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763740.990242:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763741.374179:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763741.374193:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763741.374210:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763741.374225:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3667 00000100:00080000:0.0:1789763741.374238:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763741.374248:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763741.374295:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504395136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763741.374333:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763741.374339:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763741.374341:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763741.374347:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3667 00000100:00080000:0.0:1789763741.374351:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763741.374355:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763741.374368:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763741.374382:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504395264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763741.374395:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763741.374399:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763741.374402:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763741.374406:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3667 00000100:00080000:0.0:1789763741.374410:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763741.374414:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763741.374420:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763741.374432:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504395392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763741.374446:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763741.374450:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763741.374452:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763741.374461:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504395136:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763741.374463:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3667 00000100:00080000:0.0:1789763741.374474:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763741.374483:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763741.374491:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763741.374506:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504395520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763741.374519:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763741.374523:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763741.374526:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763741.374531:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3667 00000100:00080000:0.0:1789763741.374536:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763741.374540:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763741.374546:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763741.374565:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504395648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763741.374566:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763741.374584:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763741.374590:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763741.374593:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763741.374598:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3667 00000100:00080000:0.0:1789763741.374602:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763741.374607:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763741.374614:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763741.374658:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504395776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763741.374659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763741/real 1789763741] req@ffff9018c8b98000 x1876699504395136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763741.374742:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504395136/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763741.374759:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763741.374766:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763741.374771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504395136:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763741.374782:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504395264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763741.374815:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763741.374832:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763741/real 1789763741] req@ffff9018c8b9b800 x1876699504395264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763741.374848:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504395264/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763741.374858:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763741.374862:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763741.374868:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504395264:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763741.374897:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504395392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763741.374919:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763741.374937:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763741/real 1789763741] req@ffff9018c8b9aa00 x1876699504395392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763741.374962:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504395392/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763741.374978:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763741.374984:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763741.374991:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504395392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763741.375004:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504395520:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763741.375096:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763741.375116:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763741/real 1789763741] req@ffff9018c8b98e00 x1876699504395520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763741.375133:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504395520/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763741.375144:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763741.375148:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763741.375152:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504395520:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763741.375160:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504395648:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763741.375185:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763741.375198:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763741/real 1789763741] req@ffff9018c8b9a680 x1876699504395648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763741.375211:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504395648/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763741.375221:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763741.375224:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763741.375228:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504395648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763741.375234:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504395776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763741.375252:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763741.375263:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763741/real 1789763741] req@ffff9018c8b98a80 x1876699504395776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763741.375286:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504395776/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763741.375300:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763741.375306:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763741.375312:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504395776:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763742.014290:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763743.038233:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763744.062239:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763745.086359:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763746.110238:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763746.494284:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763746.494296:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763746.494301:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763746.494313:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3673 00000100:00080000:2.0:1789763746.494326:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763746.494335:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763746.494382:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504395904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763746.494422:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763746.494427:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763746.494430:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763746.494435:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3673 00000100:00080000:2.0:1789763746.494439:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763746.494444:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763746.494473:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763746.494487:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504396032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763746.494498:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504395904:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763746.494504:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763746.494509:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763746.494511:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763746.494520:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3673 00000100:00080000:2.0:1789763746.494540:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763746.494545:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763746.494550:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763746.494568:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504396160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763746.494579:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763746.494583:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763746.494586:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763746.494610:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3673 00000100:00080000:2.0:1789763746.494616:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763746.494620:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763746.494628:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789763746.494641:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763746.494645:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504396288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763746.494660:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763746.494664:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763746.494667:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763746.494672:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3673 00000100:00080000:2.0:1789763746.494675:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763746.494681:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763746.494686:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763746.494701:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504396416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763746.494711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763746/real 1789763746] req@ffff9018d0f25c00 x1876699504395904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763746.494718:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763746.494728:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763746.494732:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763746.494747:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3673 00000100:00080000:2.0:1789763746.494775:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763746.494780:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763746.494786:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763746.494793:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504395904/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763746.494801:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504396544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763746.494839:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763746.494849:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763746.494856:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504395904:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763746.494911:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504396032:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763746.494948:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763746.494980:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763746/real 1789763746] req@ffff9018d0f26a00 x1876699504396032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763746.494992:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504396032/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763746.495001:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763746.495004:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763746.495008:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504396032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763746.495013:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504396160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763746.495052:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763746.495063:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763746/real 1789763746] req@ffff9018d0f26680 x1876699504396160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763746.495087:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504396160/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763746.495097:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763746.495100:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763746.495104:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504396160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763746.495144:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504396288:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763746.495171:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763746.495186:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763746/real 1789763746] req@ffff9018d0f24000 x1876699504396288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763746.495201:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504396288/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763746.495211:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763746.495215:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763746.495219:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504396288:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763746.495225:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504396416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763746.495259:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763746.495273:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763746/real 1789763746] req@ffff9018d0f26300 x1876699504396416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763746.495287:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504396416/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763746.495304:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763746.495308:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763746.495312:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504396416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763746.495319:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504396544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763746.495340:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763746.495351:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763746/real 1789763746] req@ffff9018d0f24380 x1876699504396544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763746.495365:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504396544/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763746.495374:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763746.495378:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763746.495381:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504396544:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763747.134331:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763748.158257:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763749.182300:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763750.206223:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763751.230397:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763751.614274:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763751.614283:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763751.614289:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763751.614301:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3678 00000100:00080000:0.0:1789763751.614315:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763751.614325:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763751.614368:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504396672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763751.614407:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763751.614411:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763751.614413:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763751.614418:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3678 00000100:00080000:0.0:1789763751.614424:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763751.614428:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763751.614439:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763751.614455:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504396800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763751.614466:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763751.614471:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763751.614473:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763751.614478:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3678 00000100:00080000:0.0:1789763751.614482:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763751.614487:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763751.614492:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763751.614530:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504396928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763751.614582:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763751.614587:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763751.614590:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763751.614615:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3678 00000100:00100000:1.0:1789763751.614622:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504396672:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763751.614624:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763751.614632:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763751.614639:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763751.614652:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504397056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763751.614662:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763751.614681:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763751.614683:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763751.614687:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3678 00000100:00080000:0.0:1789763751.614690:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763751.614693:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763751.614698:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763751.614706:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504397184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763751.614714:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763751.614717:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763751.614719:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763751.614722:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3678 00000100:00080000:0.0:1789763751.614725:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763751.614728:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763751.614732:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763751.614740:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504397312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763751.616796:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763751.616840:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763751/real 1789763751] req@ffff9018c8b98700 x1876699504396672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763751.616860:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504396672/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763751.616872:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763751.616876:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763751.616880:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504396672:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763751.616891:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504396800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763751.616913:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763751.616925:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763751/real 1789763751] req@ffff9018c8b9bb80 x1876699504396800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763751.616937:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504396800/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763751.616945:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763751.616948:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763751.616953:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504396800:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763751.616984:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504396928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763751.617016:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763751.617077:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763751/real 1789763751] req@ffff9018c8b9b100 x1876699504396928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763751.617100:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504396928/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763751.617114:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763751.617119:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763751.617124:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504396928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763751.617150:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504397056:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763751.617186:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763751.617204:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763751/real 1789763751] req@ffff9018c8b9b480 x1876699504397056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763751.617217:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504397056/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763751.617225:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763751.617229:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763751.617232:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504397056:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763751.617254:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504397184:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763751.617274:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763751.617284:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763751/real 1789763751] req@ffff9018c8b99f80 x1876699504397184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763751.617294:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504397184/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763751.617319:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763751.617322:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763751.617325:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504397184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763751.617330:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504397312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763751.617347:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763751.617357:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763751/real 1789763751] req@ffff9018c8b98a80 x1876699504397312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763751.617368:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504397312/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763751.617375:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763751.617378:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763751.617381:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504397312:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763752.254202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763753.278245:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763754.302235:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763755.326225:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763756.350223:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763756.734170:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763756.734183:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763756.734189:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763756.734202:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3683 00000100:00080000:1.0:1789763756.734233:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763756.734243:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763756.734293:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504397440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763756.734331:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763756.734336:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763756.734339:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763756.734344:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3683 00000100:00080000:1.0:1789763756.734349:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763756.734355:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763756.734368:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763756.734383:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504397568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763756.734399:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763756.734404:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504397440:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763756.734405:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763756.734412:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763756.734425:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3683 00000100:00080000:1.0:1789763756.734429:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763756.734433:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763756.734438:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763756.734458:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504397696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763756.734477:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763756.734483:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763756.734487:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763756.734497:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3683 00000100:00080000:1.0:1789763756.734505:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763756.734534:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763756.734546:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763756.734576:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504397824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763756.734597:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763756.734601:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763756.734604:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763756.734613:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504397568:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789763756.734616:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3683 00000100:00080000:1.0:1789763756.734629:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763756.734641:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763756.734651:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763756.734677:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504397952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763756.734698:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763756.734704:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789763756.734707:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504397696:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789763756.734709:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763756.734724:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3683 00000100:00080000:1.0:1789763756.734731:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763756.734735:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763756.734742:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763756.734759:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504397824:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763756.734763:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504398080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763756.734811:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504397952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763756.734839:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504398080:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763757.374231:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763758.398383:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763759.422259:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763759.806466:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:2.0:1789763759.806568:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763759.806592:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763759.806643:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763759.806653:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763759.806662:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763759.806674:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:2.0:1789763759.806682:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789763759.806718:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763756/real 1789763759] req@ffff9018c66a6300 x1876699504397952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763759.806763:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504397952/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763759.806784:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763759.806792:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763759.806798:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504397952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763759.806810:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763756/real 1789763759] req@ffff9018c66a5180 x1876699504398080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763759.806828:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504398080/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763759.806847:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763759.806853:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763759.806858:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504398080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763759.806868:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763756/real 1789763759] req@ffff9018c66a5500 x1876699504397568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763759.806897:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504397568/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763759.806910:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763759.806916:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763759.806920:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504397568:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763759.806931:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763756/real 1789763759] req@ffff9018c66a6680 x1876699504397696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763759.806950:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504397696/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763759.806962:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763759.806968:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763759.806973:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504397696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763759.806983:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763756/real 1789763759] req@ffff9018c66a4700 x1876699504397824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763759.807001:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504397824/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763759.807013:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763759.807018:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763759.807024:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504397824:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763759.807080:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763756/real 1789763759] req@ffff9018c66a4e00 x1876699504397440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763759.807101:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504397440/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763759.807113:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763759.807118:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763759.807123:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504397440:192.168.203.158@tcp:250:kworker.0 00010000:00010000:0.0:1789763760.446325:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763761.470173:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763761.854288:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763761.854306:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763761.854315:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763761.854334:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3688 00000100:00080000:2.0:1789763761.854354:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763761.854368:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763761.854444:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504398208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763761.854525:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763761.854544:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763761.854548:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763761.854565:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3688 00000100:00080000:2.0:1789763761.854574:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763761.854583:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763761.854603:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763761.854655:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504398208:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763761.854658:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504398336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763761.854693:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763761.854700:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763761.854703:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763761.854712:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3688 00000100:00080000:2.0:1789763761.854720:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763761.854728:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763761.854739:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763761.854764:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504398464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763761.854783:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763761.854791:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763761.854795:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763761.854809:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3688 00000800:00000100:0.0:1789763761.854814:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763761.854820:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763761.854831:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763761.854843:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763761.854865:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504398592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763761.854888:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763761.854896:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763761.854902:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763761.854907:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763761/real 1789763761] req@ffff9018d0f25180 x1876699504398208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763761.854916:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3688 00000100:00080000:2.0:1789763761.854944:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763761.854955:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763761.854967:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504398208/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763761.854969:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789763761.855002:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789763761.855009:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504398720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763761.855018:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763761.855090:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763761.855098:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763761.855102:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763761.855113:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3688 00000100:00080000:2.0:1789763761.855122:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763761.855132:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763761.855136:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504398208:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789763761.855145:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763761.855175:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504398848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763761.855211:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504398336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763761.855279:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763761.855308:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763761/real 1789763761] req@ffff9018d0f27100 x1876699504398336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763761.855338:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504398336/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763761.855358:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763761.855365:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763761.855372:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504398336:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763761.855386:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504398464:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763761.855430:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763761.855456:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763761/real 1789763761] req@ffff9018d0f25f80 x1876699504398464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763761.855511:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504398464/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763761.855531:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763761.855538:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763761.855545:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504398464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763761.855561:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504398592:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763761.855608:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763761.855636:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763761/real 1789763761] req@ffff9018d0f26d80 x1876699504398592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763761.855677:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504398592/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763761.855696:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763761.855705:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763761.855714:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504398592:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763761.855730:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504398720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763761.855775:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763761.855800:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763761/real 1789763761] req@ffff9018d0f24380 x1876699504398720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763761.855817:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504398720/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763761.855827:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763761.855831:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763761.855835:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504398720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763761.855843:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504398848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763761.855869:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763761.855881:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763761/real 1789763761] req@ffff9018d0f26300 x1876699504398848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763761.855896:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504398848/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763761.855919:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763761.855925:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763761.855932:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504398848:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763762.494226:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763763.518243:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763764.542247:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763765.566258:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763766.590219:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763766.974142:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763766.974151:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763766.974155:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763766.974165:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3693 00000100:00080000:0.0:1789763766.974175:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763766.974182:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763766.974216:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504398976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763766.974244:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763766.974247:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763766.974249:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763766.974252:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3693 00000100:00080000:0.0:1789763766.974255:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763766.974258:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763766.974267:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763766.974276:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504399104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763766.974285:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763766.974288:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763766.974290:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763766.974293:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3693 00000100:00080000:0.0:1789763766.974295:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763766.974298:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763766.974301:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763766.974310:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504399232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763766.974317:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763766.974320:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763766.974322:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763766.974325:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3693 00000100:00080000:0.0:1789763766.974328:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763766.974334:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504398976:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763766.974335:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763766.974344:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763766.974356:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504399360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763766.974365:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763766.974367:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763766.974369:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763766.974373:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3693 00000100:00080000:0.0:1789763766.974377:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763766.974381:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763766.974388:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763766.974402:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504399488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763766.974409:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763766.974424:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763766.974430:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763766.974432:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763766.974459:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3693 00000100:00080000:0.0:1789763766.974465:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763766.974471:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763766.974474:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763766/real 1789763766] req@ffff9018c8b9a680 x1876699504398976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763766.974479:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763766.974499:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504399616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763766.974503:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504398976/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763766.974530:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763766.974535:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763766.974540:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504398976:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763766.974547:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504399104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763766.974567:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763766.974578:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763766/real 1789763766] req@ffff9018c8b98e00 x1876699504399104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763766.974588:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504399104/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763766.974595:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763766.974598:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763766.974600:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504399104:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763766.974605:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504399232:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763766.974617:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763766.974625:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763766/real 1789763766] req@ffff9018c8b9aa00 x1876699504399232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763766.974634:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504399232/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763766.974640:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763766.974643:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763766.974645:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504399232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763766.974666:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504399360:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763766.974680:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763766.974687:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763766/real 1789763766] req@ffff9018c8b9b800 x1876699504399360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763766.974697:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504399360/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763766.974703:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763766.974705:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763766.974708:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504399360:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763766.974712:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504399488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763766.974723:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763766.974730:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763766/real 1789763766] req@ffff9018c8b98000 x1876699504399488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763766.974739:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504399488/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763766.974746:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763766.974748:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763766.974751:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504399488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763766.974755:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504399616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763766.974766:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763766.974773:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763766/real 1789763766] req@ffff9018c8b99880 x1876699504399616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763766.974781:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504399616/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763766.974786:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763766.974789:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763766.974792:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504399616:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763767.614273:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763768.638235:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763769.662260:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763770.686203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763771.710215:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763772.094229:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763772.094244:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763772.094249:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763772.094262:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3698 00000100:00080000:0.0:1789763772.094278:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763772.094288:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763772.094351:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504399744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763772.094386:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763772.094391:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763772.094394:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763772.094398:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3698 00000100:00080000:0.0:1789763772.094418:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763772.094424:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763772.094439:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763772.094455:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504399872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763772.094470:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763772.094476:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763772.094480:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763772.094483:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504399744:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763772.094488:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3698 00000100:00080000:0.0:1789763772.094496:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763772.094504:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763772.094508:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763772.094521:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504400000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763772.094530:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763772.094534:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763772.094538:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763772.094542:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3698 00000100:00080000:0.0:1789763772.094547:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763772.094551:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763772.094556:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763772.094569:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504400128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763772.094579:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763772.094587:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763772.094592:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763772.094592:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763772.094606:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3698 00000100:00080000:0.0:1789763772.094612:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763772.094617:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763772.094622:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763772.094635:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504400256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763772.094644:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763772.094649:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763772.094652:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763772.094660:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3698 00000100:00100000:1.0:1789763772.094664:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763772/real 1789763772] req@ffff9018c8b99c00 x1876699504399744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763772.094665:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763772.094677:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763772.094689:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763772.094709:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504400384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763772.094741:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504399744/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763772.094760:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763772.094768:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763772.094778:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504399744:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763772.094814:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504399872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763772.094848:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763772.094864:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763772/real 1789763772] req@ffff9018c8b9ad80 x1876699504399872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763772.094880:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504399872/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763772.094891:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763772.094896:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763772.094901:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504399872:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763772.094910:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504400000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763772.094931:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763772.094944:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763772/real 1789763772] req@ffff9018c8b9a300 x1876699504400000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763772.094959:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504400000/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763772.094970:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763772.094974:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763772.094979:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504400000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763772.094987:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504400128:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763772.095009:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763772.095024:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763772/real 1789763772] req@ffff9018c8b98380 x1876699504400128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763772.095079:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504400128/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763772.095090:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763772.095095:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763772.095100:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504400128:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763772.095108:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504400256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763772.095134:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763772.095148:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763772/real 1789763772] req@ffff9018c8b99500 x1876699504400256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763772.095162:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504400256/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763772.095172:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763772.095176:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763772.095181:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504400256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763772.095188:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504400384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763772.095211:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763772.095223:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763772/real 1789763772] req@ffff9018c8b99880 x1876699504400384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763772.095238:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504400384/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763772.095249:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763772.095254:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763772.095258:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504400384:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763772.734274:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763773.758175:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763774.782201:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763775.806214:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763776.830256:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763777.214222:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763777.214235:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763777.214240:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763777.214252:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3703 00000100:00080000:1.0:1789763777.214264:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763777.214272:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763777.214326:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504400512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763777.214358:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763777.214362:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763777.214383:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763777.214390:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3703 00000100:00080000:1.0:1789763777.214396:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763777.214429:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763777.214444:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763777.214469:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504400512:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763777.214471:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504400640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763777.214493:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763777.214497:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763777.214503:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763777.214510:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3703 00000100:00080000:1.0:1789763777.214515:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763777.214518:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763777.214524:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763777.214539:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504400768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763777.214550:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763777.214555:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763777.214557:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763777.214561:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3703 00000100:00080000:1.0:1789763777.214569:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763777.214573:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763777.214579:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789763777.214587:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763777.214594:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504400896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763777.214607:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763777.214612:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763777.214614:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763777.214618:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3703 00000100:00080000:1.0:1789763777.214621:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763777.214626:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763777.214631:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763777.214640:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763777/real 1789763777] req@ffff9018c66a7800 x1876699504400512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763777.214646:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504401024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763777.214657:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763777.214660:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763777.214662:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763777.214665:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3703 00000100:00080000:1.0:1789763777.214667:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763777.214671:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763777.214679:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763777.214692:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504401152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763777.214758:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504400512/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763777.214784:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763777.214789:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763777.214793:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504400512:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763777.214837:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504400640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763777.214867:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763777.214882:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763777/real 1789763777] req@ffff9018c66a7b80 x1876699504400640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763777.214895:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504400640/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763777.214904:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763777.214919:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763777.214923:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504400640:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763777.214951:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504400768:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763777.214972:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763777.214993:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763777/real 1789763777] req@ffff9018c66a4000 x1876699504400768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763777.215012:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504400768/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763777.215101:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763777.215105:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763777.215124:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504400768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763777.215130:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504400896:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763777.215166:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763777.215205:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763777/real 1789763777] req@ffff9018c66a5f80 x1876699504400896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763777.215229:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504400896/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763777.215238:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763777.215241:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763777.215253:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504400896:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763777.215259:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504401024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763777.215293:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763777.215305:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763777/real 1789763777] req@ffff9018c66a6d80 x1876699504401024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763777.215329:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504401024/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763777.215353:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763777.215356:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763777.215360:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504401024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763777.215401:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504401152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763777.215460:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763777.215475:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763777/real 1789763777] req@ffff9018c66a5c00 x1876699504401152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763777.215501:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504401152/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763777.215510:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763777.215514:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763777.215532:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504401152:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763777.854179:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763778.878163:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763779.902187:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763780.926155:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763781.950276:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763782.334124:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763782.334134:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763782.334138:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763782.334148:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3708 00000100:00080000:2.0:1789763782.334158:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763782.334165:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763782.334198:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504401280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763782.334225:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763782.334230:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763782.334232:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763782.334237:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3708 00000100:00080000:2.0:1789763782.334241:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763782.334246:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763782.334256:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763782.334264:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504401280:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763782.334272:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504401408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763782.334285:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763782.334289:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763782.334290:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763782.334295:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3708 00000100:00080000:2.0:1789763782.334299:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763782.334303:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763782.334307:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763782.334319:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504401536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763782.334324:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763782.334345:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763782.334350:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763782.334352:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763782.334357:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3708 00000100:00080000:2.0:1789763782.334362:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763782.334365:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763782.334372:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763782.334385:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763782/real 1789763782] req@ffff9018d0f24000 x1876699504401280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763782.334386:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504401664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763782.334413:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763782.334414:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504401280/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763782.334419:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763782.334423:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763782.334431:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763782.334432:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3708 00000100:00080000:0.0:1789763782.334439:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763782.334440:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763782.334445:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504401280:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763782.334449:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763782.334456:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763782.334468:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504401792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763782.334471:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504401408:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763782.334487:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763782.334491:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763782.334494:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763782.334495:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763782.334502:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3708 00000100:00080000:2.0:1789763782.334507:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763782.334511:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763782/real 1789763782] req@ffff9018d0f26680 x1876699504401408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763782.334515:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763782.334525:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763782.334530:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504401408/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763782.334538:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789763782.334541:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504401920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763782.334543:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763782.334550:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504401408:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763782.334561:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504401536:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763782.334579:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763782.334589:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763782/real 1789763782] req@ffff9018d0f26a00 x1876699504401536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763782.334597:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504401536/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763782.334602:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763782.334604:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763782.334605:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504401536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763782.334609:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504401664:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763782.334627:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763782.334635:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763782/real 1789763782] req@ffff9018d0f25c00 x1876699504401664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763782.334642:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504401664/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763782.334647:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763782.334650:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763782.334652:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504401664:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763782.334662:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504401792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763782.334672:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763782.334679:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763782/real 1789763782] req@ffff9018d0f27800 x1876699504401792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763782.334686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504401792/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763782.334691:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763782.334693:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763782.334695:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504401792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763782.334698:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504401920:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763782.334707:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763782.334713:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763782/real 1789763782] req@ffff9018d0f24a80 x1876699504401920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763782.334725:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504401920/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763782.334730:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763782.334731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763782.334733:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504401920:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763782.974171:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763783.998212:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763785.022220:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763786.046174:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763787.070205:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763787.454112:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763787.454122:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763787.454127:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763787.454137:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3714 00000100:00080000:2.0:1789763787.454148:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763787.454156:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763787.454196:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504402048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763787.454225:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763787.454229:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763787.454231:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763787.454237:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3714 00000100:00080000:2.0:1789763787.454240:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763787.454244:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763787.454254:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763787.454268:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504402176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763787.454280:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763787.454285:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763787.454287:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763787.454304:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3714 00000100:00080000:2.0:1789763787.454308:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763787.454314:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504402048:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763787.454316:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763787.454325:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763787.454339:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504402304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763787.454358:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763787.454362:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763787.454364:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763787.454368:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3714 00000100:00080000:2.0:1789763787.454373:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763787.454374:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763787.454381:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763787.454388:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763787.454402:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504402432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763787.454410:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763787/real 1789763787] req@ffff9018d0f24700 x1876699504402048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763787.454416:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763787.454423:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763787.454426:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763787.454429:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504402048/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763787.454434:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3714 00000100:00080000:2.0:1789763787.454443:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763787.454445:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763787.454451:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763787.454453:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763787.454457:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504402048:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789763787.454462:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763787.454465:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504402176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763787.454480:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504402560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763787.454489:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763787.454499:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763787.454503:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789763787.454504:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763787/real 1789763787] req@ffff9018d0f24e00 x1876699504402176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763787.454507:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763787.454518:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3714 00000100:00080000:2.0:1789763787.454523:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763787.454526:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504402176/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763787.454530:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763787.454538:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763787.454542:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763787.454550:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763787.454552:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504402688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763787.454558:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504402176:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763787.454592:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504402304:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763787.454610:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763787.454619:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763787/real 1789763787] req@ffff9018d0f27480 x1876699504402304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763787.454627:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504402304/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763787.454632:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763787.454635:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763787.454637:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504402304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763787.454640:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504402432:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763787.454651:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763787.454656:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763787/real 1789763787] req@ffff9018d0f25500 x1876699504402432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763787.454663:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504402432/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763787.454669:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763787.454672:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763787.454674:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504402432:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763787.454677:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504402560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763787.454686:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763787.454692:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763787/real 1789763787] req@ffff9018d0f24a80 x1876699504402560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763787.454700:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504402560/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763787.454705:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763787.454707:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763787.454709:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504402560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763787.454713:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504402688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763787.454724:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763787.454729:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763787/real 1789763787] req@ffff9018d0f27800 x1876699504402688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763787.454736:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504402688/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763787.454741:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763787.454743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763787.454746:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504402688:192.168.203.158@tcp:38:kworker.0 00010000:00010000:0.0:1789763788.094200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763789.118216:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763790.142224:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763791.166176:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763792.190153:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763792.574101:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763792.574109:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763792.574113:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763792.574121:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3719 00000100:00080000:2.0:1789763792.574130:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763792.574136:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763792.574180:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504402816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763792.574203:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763792.574217:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763792.574220:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763792.574225:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3719 00000100:00080000:2.0:1789763792.574228:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763792.574233:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763792.574242:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763792.574275:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504402816:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763792.574276:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504402944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763792.574294:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763792.574298:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763792.574300:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763792.574304:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3719 00000100:00080000:2.0:1789763792.574309:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763792.574314:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763792.574319:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763792.574333:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504403072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763792.574341:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763792.574349:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763792.574353:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763792.574355:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763792.574359:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3719 00000100:00080000:2.0:1789763792.574362:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763792.574366:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763792.574370:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763792.574380:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763792/real 1789763792] req@ffff9018d0f25c00 x1876699504402816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763792.574384:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504403200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763792.574408:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763792.574410:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763792.574412:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763792.574415:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504402816/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763792.574417:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3719 00000100:00080000:2.0:1789763792.574422:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763792.574429:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763792.574435:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789763792.574435:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763792.574443:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763792.574447:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504403328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763792.574450:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504402816:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763792.574462:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763792.574465:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763792.574466:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763792.574470:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3719 00000100:00080000:2.0:1789763792.574471:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763792.574474:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763792.574477:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763792.574480:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504402944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763792.574487:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504403456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763792.574514:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763792.574524:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763792/real 1789763792] req@ffff9018d0f26a00 x1876699504402944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763792.574533:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504402944/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763792.574539:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763792.574541:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763792.574544:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504402944:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763792.574547:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504403072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763792.574557:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763792.574564:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763792/real 1789763792] req@ffff9018d0f26680 x1876699504403072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763792.574571:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504403072/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763792.574577:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763792.574578:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763792.574580:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504403072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763792.574584:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504403200:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763792.574594:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763792.574600:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763792/real 1789763792] req@ffff9018d0f24000 x1876699504403200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763792.574608:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504403200/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763792.574613:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763792.574614:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763792.574616:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504403200:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763792.574619:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504403328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763792.574629:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763792.574634:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763792/real 1789763792] req@ffff9018d0f26300 x1876699504403328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763792.574640:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504403328/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763792.574647:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763792.574649:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763792.574651:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504403328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763792.574661:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504403456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763792.574670:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763792.574676:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763792/real 1789763792] req@ffff9018d0f24380 x1876699504403456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763792.574683:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504403456/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763792.574689:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763792.574690:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763792.574692:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504403456:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763793.214182:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763794.238154:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763795.262148:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763796.286252:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763797.310152:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763797.694104:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763797.694114:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763797.694118:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763797.694129:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3724 00000100:00080000:2.0:1789763797.694139:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763797.694146:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763797.694180:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504403584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763797.694209:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763797.694225:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763797.694228:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763797.694233:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3724 00000100:00080000:2.0:1789763797.694238:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763797.694242:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763797.694254:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504403584:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789763797.694254:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763797.694273:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504403712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763797.694286:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763797.694290:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763797.694292:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763797.694296:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3724 00000100:00080000:2.0:1789763797.694302:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763797.694306:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789763797.694311:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789763797.694315:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763797.694330:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504403840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763797.694342:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763797.694345:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763797/real 1789763797] req@ffff9018d0f26d80 x1876699504403584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763797.694350:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763797.694355:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763797.694363:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3724 00000100:00100000:1.0:1789763797.694368:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504403584/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763797.694369:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763797.694379:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763797.694381:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789763797.694385:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1789763797.694386:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763797.694393:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504403584:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763797.694403:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504403968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763797.694414:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504403712:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763797.694419:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763797.694424:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763797.694427:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763797.694432:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3724 00000800:00000100:1.0:1789763797.694434:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763797.694440:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763797.694447:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763797.694449:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763797/real 1789763797] req@ffff9018d0f25f80 x1876699504403712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763797.694454:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763797.694464:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504403712/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763797.694473:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789763797.694474:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504404096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763797.694479:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763797.694485:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504403712:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763797.694495:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504403840:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763797.694498:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763797.694503:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763797.694506:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763797.694511:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763797.694512:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3724 00000100:00080000:2.0:1789763797.694520:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763797.694523:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763797/real 1789763797] req@ffff9018d0f27100 x1876699504403840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763797.694528:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763797.694537:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763797.694546:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504403840/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763797.694552:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504404224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763797.694563:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763797.694573:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763797.694578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504403840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763797.694596:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504403968:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763797.694617:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763797.694629:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763797/real 1789763797] req@ffff9018d0f25180 x1876699504403968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763797.694642:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504403968/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763797.694650:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763797.694653:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763797.694657:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504403968:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763797.694672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504404096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763797.694693:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763797.694705:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763797/real 1789763797] req@ffff9018d0f26d80 x1876699504404096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763797.694719:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504404096/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763797.694728:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763797.694731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763797.694733:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504404096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763797.694738:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504404224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763797.694753:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763797.694760:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763797/real 1789763797] req@ffff9018d0f24380 x1876699504404224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763797.694767:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504404224/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763797.694774:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763797.694776:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763797.694778:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504404224:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763798.334153:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763799.358142:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763800.382160:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763801.406138:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763802.430147:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763802.814135:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763802.814144:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763802.814147:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763802.814155:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3729 00000100:00080000:3.0:1789763802.814163:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763802.814168:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763802.814223:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504404352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763802.814249:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763802.814253:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763802.814255:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763802.814260:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3729 00000100:00080000:3.0:1789763802.814266:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763802.814272:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763802.814282:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763802.814289:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504404352:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763802.814300:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504404480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763802.814314:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763802.814319:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763802.814322:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763802.814327:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3729 00000100:00080000:3.0:1789763802.814331:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763802.814336:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763802.814341:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763802.814355:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763802.814356:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504404608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763802.814367:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763802.814369:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763802.814371:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763802.814374:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3729 00000100:00080000:3.0:1789763802.814388:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763802.814393:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763802.814400:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763802.814403:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763802/real 1789763802] req@ffff9018d070ca80 x1876699504404352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763802.814416:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504404736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763802.814424:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504404352/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763802.814436:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763802.814441:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763802.814442:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763802.814447:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763802.814449:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789763802.814457:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3729 00000100:00100000:1.0:1789763802.814458:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504404352:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763802.814466:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763802.814480:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763802.814486:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763802.814488:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504404480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763802.814501:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504404864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763802.814511:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763802.814515:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1789763802.814518:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763802.814519:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763802.814526:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3729 00000100:00080000:3.0:1789763802.814531:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763802.814535:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763802.814540:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763802/real 1789763802] req@ffff9018d070d180 x1876699504404480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789763802.814541:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763802.814554:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504404992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763802.814561:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504404480/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763802.814571:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763802.814575:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763802.814580:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504404480:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763802.814588:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504404608:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763802.814605:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763802.814615:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763802/real 1789763802] req@ffff9018d070e680 x1876699504404608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763802.814626:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504404608/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763802.814634:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763802.814637:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763802.814640:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504404608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763802.814646:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504404736:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763802.814661:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763802.814670:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763802/real 1789763802] req@ffff9018d070d500 x1876699504404736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763802.814680:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504404736/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763802.814688:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763802.814691:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763802.814694:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504404736:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763802.814708:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504404864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763802.814724:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763802.814733:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763802/real 1789763802] req@ffff9018d070dc00 x1876699504404864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763802.814744:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504404864/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763802.814752:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763802.814754:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763802.814757:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504404864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763802.814763:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504404992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763802.814778:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763802.814788:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763802/real 1789763802] req@ffff9018d070c000 x1876699504404992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763802.814800:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504404992/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763802.814807:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763802.814811:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763802.814814:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504404992:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763803.454213:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763804.478152:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763805.502221:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763806.526142:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763807.550198:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763807.934118:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763807.934126:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763807.934129:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763807.934156:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3734 00000100:00080000:0.0:1789763807.934166:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763807.934171:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763807.934202:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504405120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763807.934225:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763807.934228:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763807.934229:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763807.934232:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3734 00000100:00080000:0.0:1789763807.934234:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763807.934236:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763807.934243:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763807.934253:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504405248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763807.934261:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763807.934263:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763807.934264:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763807.934267:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3734 00000100:00080000:0.0:1789763807.934269:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763807.934274:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763807.934274:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504405120:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789763807.934281:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763807.934287:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504405376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763807.934293:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763807.934296:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763807.934297:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763807.934300:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3734 00000100:00080000:0.0:1789763807.934302:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763807.934305:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763807.934308:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763807.934315:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504405504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763807.934321:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763807.934323:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763807.934325:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763807.934327:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3734 00000100:00080000:0.0:1789763807.934330:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763807.934335:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789763807.934337:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789763807.934339:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763807.934348:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504405632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763807.934355:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763807.934357:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763807.934358:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763807.934360:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3734 00000100:00080000:0.0:1789763807.934363:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763807.934366:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763807.934369:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763807.934373:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763807/real 1789763807] req@ffff9018c8b98000 x1876699504405120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763807.934380:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504405760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763807.934403:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504405120/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763807.934417:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763807.934420:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763807.934423:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504405120:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763807.934428:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504405248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763807.934452:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763807.934462:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763807/real 1789763807] req@ffff9018c8b9b800 x1876699504405248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763807.934475:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504405248/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763807.934484:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763807.934488:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763807.934491:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504405248:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763807.934515:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504405376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763807.934534:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763807.934543:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763807/real 1789763807] req@ffff9018c8b9aa00 x1876699504405376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763807.934553:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504405376/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763807.934562:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763807.934566:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763807.934570:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504405376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763807.934576:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504405504:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763807.934597:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763807.934608:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763807/real 1789763807] req@ffff9018c8b98e00 x1876699504405504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763807.934626:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504405504/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763807.934632:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763807.934635:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763807.934637:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504405504:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763807.934641:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504405632:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763807.934656:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763807.934663:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763807/real 1789763807] req@ffff9018c8b9a680 x1876699504405632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763807.934670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504405632/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763807.934675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763807.934677:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763807.934679:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504405632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763807.934683:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504405760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763807.934693:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763807.934698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763807/real 1789763807] req@ffff9018c8b98a80 x1876699504405760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763807.934711:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504405760/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763807.934720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763807.934724:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763807.934727:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504405760:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763808.574199:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763809.598162:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763810.622267:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763811.646230:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763812.670241:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763813.054248:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763813.054256:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763813.054260:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763813.054268:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3739 00000100:00080000:1.0:1789763813.054278:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763813.054285:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763813.054318:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504405888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763813.054353:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763813.054356:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763813.054357:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763813.054360:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3739 00000100:00080000:1.0:1789763813.054363:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763813.054366:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763813.054374:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763813.054385:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504405888:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763813.054386:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504406016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763813.054404:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763813.054406:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763813.054408:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763813.054411:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3739 00000100:00080000:1.0:1789763813.054414:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763813.054416:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763813.054419:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763813.054427:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504406144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763813.054434:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763813.054437:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763813.054438:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763813.054441:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3739 00000100:00080000:1.0:1789763813.054444:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763813.054447:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763813.054451:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763813.054456:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763813.054465:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504406272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763813.054472:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763813.054475:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763813.054476:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763813.054480:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3739 00000100:00080000:1.0:1789763813.054484:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763813.054487:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763813/real 1789763813] req@ffff9018c66a5c00 x1876699504405888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763813.054491:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763813.054495:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763813.054504:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504406400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763813.054515:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763813.054520:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504405888/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763813.054520:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763813.054524:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763813.054531:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3739 00000100:00080000:1.0:1789763813.054536:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763813.054537:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763813.054540:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:0.0:1789763813.054544:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000008:00100000:1.0:1789763813.054544:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763813.054551:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504405888:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763813.054555:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504406528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763813.054574:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504406016:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763813.054591:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763813.054600:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763813/real 1789763813] req@ffff9018c66a6d80 x1876699504406016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763813.054608:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504406016/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763813.054613:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763813.054615:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763813.054617:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504406016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763813.054621:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504406144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763813.054650:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763813.054657:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763813/real 1789763813] req@ffff9018c66a5f80 x1876699504406144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763813.054665:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504406144/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763813.054671:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763813.054673:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763813.054675:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504406144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763813.054679:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504406272:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763813.054691:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763813.054698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763813/real 1789763813] req@ffff9018c66a4000 x1876699504406272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763813.054707:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504406272/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763813.054713:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763813.054715:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763813.054717:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504406272:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763813.054721:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504406400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763813.054733:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763813.054740:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763813/real 1789763813] req@ffff9018c66a7b80 x1876699504406400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763813.054748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504406400/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763813.054754:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763813.054756:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763813.054758:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504406400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763813.054762:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504406528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763813.054773:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763813.054779:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763813/real 1789763813] req@ffff9018c66a7800 x1876699504406528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763813.054787:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504406528/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763813.054810:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763813.054814:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763813.054818:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504406528:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763813.694230:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763814.718320:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763815.742200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763816.766236:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763817.790303:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763818.174219:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763818.174231:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763818.174235:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763818.174245:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3744 00000100:00080000:3.0:1789763818.174258:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763818.174266:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763818.174307:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504406656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763818.174361:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763818.174366:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763818.174369:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763818.174374:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3744 00000100:00080000:3.0:1789763818.174378:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763818.174383:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763818.174396:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763818.174411:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504406784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763818.174426:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763818.174431:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763818.174433:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763818.174440:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3744 00000100:00080000:3.0:1789763818.174445:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763818.174449:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763818.174455:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763818.174472:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504406912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763818.174475:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504406656:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763818.174501:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763818.174506:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763818.174508:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763818.174515:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3744 00000100:00080000:3.0:1789763818.174520:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763818.174524:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763818.174531:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763818.174550:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504407040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763818.174563:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763818.174567:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763818.174569:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763818.174577:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:3.0:1789763818.174579:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3744 00000100:00080000:3.0:1789763818.174589:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763818.174594:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763818.174601:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763818.174616:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504407168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763818.174629:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763818.174634:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763818.174637:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763818.174645:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3744 00000100:00100000:0.0:1789763818.174646:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763818/real 1789763818] req@ffff9018d070f800 x1876699504406656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763818.174650:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763818.174661:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763818.174675:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763818.174699:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504406656/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763818.174701:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504407296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763818.174720:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763818.174737:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763818.174753:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504406656:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763818.174766:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504406784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763818.174797:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763818.174813:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763818/real 1789763818] req@ffff9018d070f100 x1876699504406784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763818.174828:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504406784/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763818.174838:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763818.174842:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763818.174846:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504406784:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763818.174875:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504406912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763818.174896:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763818.174908:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763818/real 1789763818] req@ffff9018d070e300 x1876699504406912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763818.174922:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504406912/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763818.174949:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763818.174952:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763818.174955:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504406912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763818.174961:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504407040:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763818.174978:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763818.174988:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763818/real 1789763818] req@ffff9018d070c000 x1876699504407040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763818.174999:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504407040/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763818.175007:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763818.175010:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763818.175012:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504407040:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763818.175018:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504407168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763818.175031:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763818.175041:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763818/real 1789763818] req@ffff9018d070dc00 x1876699504407168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763818.175117:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504407168/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763818.175126:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763818.175129:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763818.175132:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504407168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763818.175139:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504407296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763818.175160:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763818.175171:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763818/real 1789763818] req@ffff9018d070d500 x1876699504407296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763818.175183:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504407296/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763818.175191:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763818.175193:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763818.175196:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504407296:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763818.814296:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763819.838344:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763820.862235:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763821.886188:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763822.910170:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763823.294097:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763823.294105:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763823.294109:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763823.294118:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3749 00000100:00080000:3.0:1789763823.294127:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763823.294134:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763823.294162:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504407424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763823.294186:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763823.294189:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763823.294190:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763823.294193:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3749 00000100:00080000:3.0:1789763823.294196:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763823.294198:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763823.294206:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763823.294217:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504407552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763823.294224:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763823.294228:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763823.294229:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763823.294232:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3749 00000100:00080000:3.0:1789763823.294236:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763823.294241:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763823.294243:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504407424:192.168.203.158@tcp:250:kworker.0 00000008:00100000:3.0:1789763823.294245:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763823.294258:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504407680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763823.294266:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763823.294268:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763823.294270:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763823.294273:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3749 00000100:00080000:3.0:1789763823.294275:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763823.294278:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763823.294281:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763823.294290:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504407808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763823.294297:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763823.294300:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763823.294302:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763823.294304:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3749 00000100:00100000:1.0:1789763823.294344:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504407552:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789763823.294377:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763823.294382:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763823.294387:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763823.294388:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504407680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763823.294401:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504407936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763823.294412:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763823.294415:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763823.294417:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763823.294420:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3749 00000100:00080000:3.0:1789763823.294424:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763823.294427:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504407808:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763823.294471:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504407936:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789763823.294526:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763823.294531:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763823.294539:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504408064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763823.294597:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504408064:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763823.934165:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763824.958229:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763825.982185:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763826.366374:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:1.0:1789763826.366423:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789763826.366440:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763826.366475:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763826.366485:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763826.366494:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:1.0:1789763826.366502:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789763826.366508:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763823/real 1789763826] req@ffff9018d070ca80 x1876699504407680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1789763826.366510:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:0.0:1789763826.366532:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504407680/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763826.366546:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763826.366551:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763826.366557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504407680:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763826.366566:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763823/real 1789763826] req@ffff9018d070df80 x1876699504407808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763826.366578:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504407808/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763826.366587:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763826.366590:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763826.366594:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504407808:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763826.366611:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763823/real 1789763826] req@ffff9018d070e680 x1876699504407424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763826.366624:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504407424/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763826.366631:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763826.366635:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763826.366638:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504407424:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763826.366651:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763823/real 1789763826] req@ffff9018d070d180 x1876699504407552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763826.366663:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504407552/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763826.366671:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763826.366674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763826.366677:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504407552:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763826.366703:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763823/real 1789763826] req@ffff9018d070ce00 x1876699504408064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763826.366716:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504408064/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763826.366724:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763826.366728:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763826.366731:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504408064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763826.366737:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763823/real 1789763826] req@ffff9018d070ed80 x1876699504407936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763826.366749:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504407936/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763826.366757:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763826.366760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763826.366764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504407936:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763827.006219:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763828.030148:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763828.414245:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763828.414257:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763828.414261:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763828.414274:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3754 00000100:00080000:0.0:1789763828.414285:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763828.414293:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763828.414331:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504408192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763828.414363:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763828.414367:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763828.414369:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763828.414374:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3754 00000100:00080000:0.0:1789763828.414379:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763828.414384:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763828.414393:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763828.414407:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504408320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763828.414426:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763828.414430:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763828.414432:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763828.414437:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3754 00000100:00080000:0.0:1789763828.414442:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763828.414444:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504408192:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763828.414449:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763828.414459:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763828.414474:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504408448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763828.414494:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763828.414499:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763828.414501:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763828.414505:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3754 00000100:00080000:0.0:1789763828.414511:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763828.414515:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763828.414523:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763828.414523:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763828.414541:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504408576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763828.414553:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763828.414557:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763828.414559:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763828.414565:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763828/real 1789763828] req@ffff9018c8b99f80 x1876699504408192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763828.414567:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3754 00000100:00080000:0.0:1789763828.414575:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763828.414583:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763828.414591:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763828.414601:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504408192/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763828.414616:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504408704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763828.414618:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763828.414629:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789763828.414639:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763828.414640:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504408192:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763828.414648:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763828.414652:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763828.414657:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504408320:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763828.414660:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3754 00000100:00080000:0.0:1789763828.414669:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763828.414674:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763828.414681:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763828.414697:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763828.414700:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504408832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763828.414718:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763828/real 1789763828] req@ffff9018c8b9b480 x1876699504408320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763828.414737:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504408320/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763828.414755:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763828.414760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763828.414765:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504408320:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763828.414794:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504408448:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763828.414819:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763828.414832:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763828/real 1789763828] req@ffff9018c8b9b100 x1876699504408448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763828.414846:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504408448/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763828.414855:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763828.414859:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763828.414862:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504408448:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763828.414869:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504408576:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763828.414891:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763828.414905:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763828/real 1789763828] req@ffff9018c8b9bb80 x1876699504408576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763828.414920:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504408576/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763828.414952:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763828.414956:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763828.414961:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504408576:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763828.414970:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504408704:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763828.415016:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763828.415030:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763828/real 1789763828] req@ffff9018c8b98700 x1876699504408704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763828.415079:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504408704/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763828.415088:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763828.415092:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763828.415096:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504408704:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763828.415104:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504408832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763828.415130:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763828.415148:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763828/real 1789763828] req@ffff9018c8b98a80 x1876699504408832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763828.415163:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504408832/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763828.415174:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763828.415179:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763828.415183:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504408832:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763829.054276:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763830.078257:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763831.102169:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763832.126190:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763833.150296:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763833.534149:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763833.534161:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763833.534166:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763833.534177:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3760 00000100:00080000:0.0:1789763833.534189:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763833.534197:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763833.534240:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504408960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763833.534273:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763833.534278:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763833.534280:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763833.534284:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3760 00000100:00080000:0.0:1789763833.534288:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763833.534291:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763833.534302:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763833.534315:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504409088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763833.534326:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763833.534345:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763833.534346:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763833.534348:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3760 00000100:00080000:0.0:1789763833.534351:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763833.534354:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763833.534358:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763833.534367:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504409216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763833.534375:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763833.534380:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763833.534380:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504408960:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763833.534385:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763833.534392:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3760 00000100:00080000:0.0:1789763833.534411:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763833.534415:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763833.534421:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763833.534435:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504409344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763833.534446:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763833.534450:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763833.534453:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763833.534456:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3760 00000100:00080000:0.0:1789763833.534462:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763833.534467:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763833.534471:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763833.534481:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763833.534488:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504409472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763833.534502:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763833.534506:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763833.534509:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763833.534513:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3760 00000100:00080000:0.0:1789763833.534517:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763833.534523:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763833.534528:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763833.534539:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763833/real 1789763833] req@ffff9018c8b9a680 x1876699504408960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763833.534545:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504409600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763833.534597:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504408960/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763833.534613:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763833.534619:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763833.534639:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504408960:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763833.534645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504409088:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763833.534663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763833.534671:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763833/real 1789763833] req@ffff9018c8b98e00 x1876699504409088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763833.534680:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504409088/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763833.534686:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763833.534688:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763833.534691:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504409088:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763833.534708:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504409216:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763833.534720:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763833.534726:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763833/real 1789763833] req@ffff9018c8b9aa00 x1876699504409216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763833.534735:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504409216/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763833.534746:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763833.534748:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763833.534750:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504409216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763833.534754:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504409344:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763833.534786:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763833.534796:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763833/real 1789763833] req@ffff9018c8b9b800 x1876699504409344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763833.534807:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504409344/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763833.534816:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763833.534820:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763833.534823:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504409344:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763833.534847:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504409472:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763833.534868:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763833.534878:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763833/real 1789763833] req@ffff9018c8b98000 x1876699504409472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763833.534893:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504409472/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763833.534902:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763833.534905:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763833.534910:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504409472:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763833.534916:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504409600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763833.534933:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763833.534943:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763833/real 1789763833] req@ffff9018c8b99880 x1876699504409600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763833.534986:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504409600/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763833.535012:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763833.535014:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763833.535016:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504409600:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763834.174200:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763835.198231:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763836.222217:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763837.246249:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763838.270225:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763838.654212:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763838.654230:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763838.654238:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763838.654273:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3765 00000100:00080000:1.0:1789763838.654292:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763838.654304:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763838.654364:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504409728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763838.654409:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763838.654415:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763838.654418:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763838.654425:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3765 00000100:00080000:1.0:1789763838.654431:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763838.654439:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763838.654456:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763838.654461:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504409728:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763838.654480:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504409856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763838.654500:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763838.654506:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763838.654525:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763838.654530:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3765 00000100:00080000:1.0:1789763838.654536:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763838.654551:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763838.654560:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763838.654578:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504409984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763838.654592:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763838.654599:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1789763838.654602:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763838.654603:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763838.654613:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3765 00000100:00080000:1.0:1789763838.654620:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763838.654626:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763838.654634:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763838.654650:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504410112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763838.654666:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763838.654673:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763838/real 1789763838] req@ffff9018c66a4e00 x1876699504409728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763838.654675:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763838.654681:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763838.654695:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3765 00000100:00080000:1.0:1789763838.654709:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763838.654715:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763838.654719:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504409728/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789763838.654724:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789763838.654737:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763838.654743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763838.654753:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504409728:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763838.654755:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504410240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763838.654773:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763838.654778:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763838.654781:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763838.654787:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3765 00000100:00100000:0.0:1789763838.654793:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504409856:192.168.203.158@tcp:38:kworker.0 00000100:00080000:1.0:1789763838.654794:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763838.654806:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763838.654813:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763838.654829:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504410368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763838.654830:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763838.654848:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763838/real 1789763838] req@ffff9018c66a4700 x1876699504409856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763838.654863:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504409856/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763838.654878:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763838.654883:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763838.654886:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504409856:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763838.654891:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504409984:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763838.654907:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763838.654950:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763838/real 1789763838] req@ffff9018c66a6680 x1876699504409984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763838.654980:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504409984/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763838.654988:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763838.654991:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763838.654995:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504409984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763838.655000:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504410112:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763838.655027:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763838.655037:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763838/real 1789763838] req@ffff9018c66a5500 x1876699504410112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763838.655075:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504410112/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763838.655084:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763838.655087:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763838.655092:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504410112:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763838.655098:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504410240:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763838.655118:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763838.655129:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763838/real 1789763838] req@ffff9018c66a5180 x1876699504410240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763838.655141:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504410240/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763838.655149:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763838.655152:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763838.655156:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504410240:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763838.655175:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504410368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763838.655190:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763838.655200:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763838/real 1789763838] req@ffff9018c66a6300 x1876699504410368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763838.655211:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504410368/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763838.655218:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763838.655221:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763838.655224:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504410368:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763839.294212:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763840.318273:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763841.342229:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763842.366281:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763843.390245:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763843.774150:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763843.774163:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763843.774168:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763843.774180:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3770 00000100:00080000:1.0:1789763843.774193:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763843.774203:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763843.774264:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504410496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763843.774300:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763843.774305:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763843.774307:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763843.774312:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3770 00000100:00080000:1.0:1789763843.774316:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763843.774320:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763843.774332:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763843.774346:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504410624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763843.774368:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763843.774372:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763843.774375:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763843.774379:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3770 00000100:00080000:1.0:1789763843.774383:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763843.774388:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763843.774392:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763843.774402:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504410752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763843.774414:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763843.774418:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763843.774421:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763843.774425:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3770 00000100:00080000:1.0:1789763843.774432:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763843.774437:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763843.774444:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763843.774454:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504410880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763843.774466:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763843.774470:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763843.774472:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763843.774488:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3770 00000100:00100000:0.0:1789763843.774492:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504410496:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763843.774494:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763843.774514:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763843.774521:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763843.774540:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504411008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763843.774552:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763843.774558:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763843.774560:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763843.774565:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3770 00000100:00080000:1.0:1789763843.774570:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763843.774575:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763843.774581:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763843.774599:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504411136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763843.774637:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763843.774739:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763843/real 1789763843] req@ffff9018c66a5880 x1876699504410496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763843.774768:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504410496/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763843.774792:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763843.774799:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763843.774805:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504410496:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763843.774819:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504410624:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763843.774853:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763843.774870:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763843/real 1789763843] req@ffff9018c66a7480 x1876699504410624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763843.774934:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504410624/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763843.774946:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763843.774951:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763843.774955:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504410624:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763843.774967:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504410752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763843.775010:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763843.775101:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763843/real 1789763843] req@ffff9018c66a4a80 x1876699504410752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763843.775123:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504410752/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763843.775143:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763843.775147:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763843.775152:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504410752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763843.775199:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504410880:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763843.775239:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763843.775253:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763843/real 1789763843] req@ffff9018c66a7100 x1876699504410880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763843.775268:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504410880/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763843.775279:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763843.775283:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763843.775287:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504410880:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763843.775294:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504411008:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763843.775315:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763843.775326:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763843/real 1789763843] req@ffff9018c66a4380 x1876699504411008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763843.775340:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504411008/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763843.775349:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763843.775353:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763843.775357:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504411008:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763843.775364:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504411136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763843.775383:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763843.775395:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763843/real 1789763843] req@ffff9018c66a6a00 x1876699504411136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763843.775408:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504411136/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763843.775417:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763843.775420:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763843.775424:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504411136:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763844.414276:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763845.438332:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763846.462214:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763847.486277:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763848.510284:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763848.894173:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763848.894189:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763848.894195:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763848.894210:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3775 00000100:00080000:2.0:1789763848.894227:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763848.894237:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763848.894300:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504411264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763848.894344:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763848.894366:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763848.894370:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763848.894378:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3775 00000100:00080000:2.0:1789763848.894383:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763848.894389:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763848.894404:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763848.894416:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504411264:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763848.894428:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504411392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763848.894446:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763848.894451:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763848.894454:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763848.894461:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3775 00000100:00080000:2.0:1789763848.894467:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763848.894472:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763848.894478:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763848.894492:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504411520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763848.894503:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763848.894521:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763848.894527:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763848.894530:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763848.894537:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3775 00000100:00080000:2.0:1789763848.894542:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763848.894549:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763848.894557:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763848.894580:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504411648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763848.894594:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763848/real 1789763848] req@ffff9018d0f26300 x1876699504411264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763848.894596:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763848.894602:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763848.894605:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763848.894633:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3775 00000100:00080000:2.0:1789763848.894637:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763848.894648:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763848.894662:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763848.894677:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504411776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763848.894686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504411264/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763848.894717:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763848.894725:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763848.894726:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763848.894729:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763848.894736:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763848.894743:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3775 00000100:00080000:2.0:1789763848.894748:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763848.894754:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504411264:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763848.894755:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763848.894775:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763848.894794:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504411904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763848.894802:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504411392:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763848.894877:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763848.894898:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763848/real 1789763848] req@ffff9018d0f24000 x1876699504411392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763848.894908:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504411392/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763848.894915:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763848.894918:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763848.894920:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504411392:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763848.894927:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504411520:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763848.894949:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763848.894963:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763848/real 1789763848] req@ffff9018d0f26680 x1876699504411520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763848.894977:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504411520/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763848.894987:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763848.894989:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763848.894992:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504411520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763848.894997:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504411648:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763848.895015:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763848.895023:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763848/real 1789763848] req@ffff9018d0f26a00 x1876699504411648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763848.895061:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504411648/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763848.895068:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763848.895071:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763848.895073:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504411648:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763848.895078:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504411776:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763848.895093:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763848.895101:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763848/real 1789763848] req@ffff9018d0f25c00 x1876699504411776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763848.895109:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504411776/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763848.895115:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763848.895117:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763848.895119:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504411776:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763848.895134:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504411904:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763848.895146:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763848.895153:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763848/real 1789763848] req@ffff9018d0f27800 x1876699504411904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763848.895160:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504411904/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763848.895166:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763848.895168:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763848.895171:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504411904:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763849.534222:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763850.558242:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763851.582229:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763852.606264:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763853.630221:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763854.014291:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763854.014312:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763854.014320:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763854.014341:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3780 00000100:00080000:1.0:1789763854.014363:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763854.014379:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763854.014462:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504412032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763854.014524:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763854.014532:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763854.014536:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763854.014544:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3780 00000100:00080000:1.0:1789763854.014551:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763854.014560:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763854.014581:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763854.014609:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504412160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763854.014635:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504412032:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763854.014639:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763854.014654:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763854.014659:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763854.014666:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3780 00000100:00080000:1.0:1789763854.014672:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763854.014678:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763854.014688:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763854.014706:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504412288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763854.014720:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763854.014725:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763854.014728:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763854.014733:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3780 00000100:00080000:1.0:1789763854.014751:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763854.014757:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763854.014764:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789763854.014780:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763854.014782:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504412416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763854.014821:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763854.014827:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763854.014830:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763854.014837:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3780 00000100:00080000:1.0:1789763854.014843:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763854.014850:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763854.014857:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763854.014876:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504412544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763854.014890:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763854.014898:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763854/real 1789763854] req@ffff9018c66a6a00 x1876699504412032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763854.014898:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763854.014905:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763854.014920:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3780 00000100:00080000:1.0:1789763854.014930:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763854.014935:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763854.014941:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763854.014948:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504412032/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763854.014962:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504412672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763854.014972:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763854.014985:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763854.014994:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504412032:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763854.015079:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504412160:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763854.015115:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763854.015135:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763854/real 1789763854] req@ffff9018c66a4380 x1876699504412160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763854.015152:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504412160/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763854.015163:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763854.015167:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763854.015171:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504412160:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763854.015179:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504412288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763854.015201:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763854.015213:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763854/real 1789763854] req@ffff9018c66a7100 x1876699504412288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763854.015228:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504412288/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763854.015237:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763854.015241:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763854.015245:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504412288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763854.015252:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504412416:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763854.015288:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763854.015303:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763854/real 1789763854] req@ffff9018c66a4a80 x1876699504412416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763854.015327:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504412416/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763854.015343:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763854.015350:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763854.015356:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504412416:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763854.015368:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504412544:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763854.015410:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763854.015433:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763854/real 1789763854] req@ffff9018c66a7480 x1876699504412544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763854.015449:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504412544/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763854.015460:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763854.015464:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763854.015468:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504412544:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763854.015475:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504412672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763854.015500:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763854.015511:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763854/real 1789763854] req@ffff9018c66a5880 x1876699504412672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763854.015525:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504412672/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763854.015535:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763854.015538:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763854.015542:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504412672:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763854.654221:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763855.678236:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763856.702323:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763857.726268:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763858.750240:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763859.134186:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763859.134195:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763859.134200:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763859.134211:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3785 00000100:00080000:2.0:1789763859.134224:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763859.134231:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763859.134273:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504412800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763859.134299:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763859.134303:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763859.134304:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763859.134309:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3785 00000100:00080000:2.0:1789763859.134312:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763859.134315:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763859.134325:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504412800:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789763859.134326:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763859.134342:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504412928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763859.134352:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763859.134355:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763859.134357:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763859.134361:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3785 00000100:00080000:2.0:1789763859.134365:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763859.134368:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763859.134373:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763859.134382:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504413056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763859.134389:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763859.134392:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763859.134394:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763859.134396:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3785 00000100:00080000:2.0:1789763859.134399:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763859.134402:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:1.0:1789763859.134407:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789763859.134408:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763859.134420:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504413184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763859.134428:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763859.134431:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763859.134433:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763859.134436:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3785 00000100:00080000:2.0:1789763859.134439:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763859.134442:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763859.134446:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763859.134448:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763859/real 1789763859] req@ffff9018d0f24a80 x1876699504412800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763859.134457:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504413312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763859.134467:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763859.134468:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504412800/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763859.134477:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763859.134481:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763859.134486:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763859.134487:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3785 00000100:00080000:2.0:1789763859.134492:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763859.134494:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763859.134498:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763859.134502:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504412800:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789763859.134504:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763859.134515:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504413440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763859.134530:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504412928:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763859.134550:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763859.134561:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763859/real 1789763859] req@ffff9018d0f25500 x1876699504412928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763859.134570:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504412928/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763859.134580:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763859.134583:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763859.134586:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504412928:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763859.134591:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504413056:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763859.134605:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763859.134611:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763859/real 1789763859] req@ffff9018d0f27480 x1876699504413056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763859.134619:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504413056/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763859.134625:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763859.134627:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763859.134629:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504413056:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763859.134633:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504413184:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763859.134644:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763859.134650:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763859/real 1789763859] req@ffff9018d0f24e00 x1876699504413184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763859.134657:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504413184/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763859.134663:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763859.134665:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763859.134667:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504413184:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763859.134671:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504413312:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763859.134683:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763859.134688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763859/real 1789763859] req@ffff9018d0f24700 x1876699504413312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763859.134695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504413312/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763859.134701:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763859.134703:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763859.134705:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504413312:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763859.134708:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504413440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763859.134722:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763859.134728:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763859/real 1789763859] req@ffff9018d0f27800 x1876699504413440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763859.134736:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504413440/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763859.134741:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763859.134743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763859.134745:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504413440:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763859.774241:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763860.798260:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763861.822262:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763862.846201:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763863.870234:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763864.254150:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763864.254163:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763864.254169:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763864.254182:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3790 00000100:00080000:2.0:1789763864.254197:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763864.254206:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763864.254254:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504413568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763864.254303:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763864.254309:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763864.254312:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763864.254318:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3790 00000100:00080000:2.0:1789763864.254323:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763864.254330:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763864.254343:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763864.254366:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504413696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763864.254381:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504413568:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763864.254397:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763864.254407:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763864.254411:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763864.254423:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3790 00000100:00080000:2.0:1789763864.254431:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763864.254441:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763864.254452:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763864.254479:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504413824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763864.254501:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763864.254501:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763864.254516:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763864.254522:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763864.254533:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3790 00000100:00080000:2.0:1789763864.254543:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763864.254553:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763864.254565:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763864/real 1789763864] req@ffff9018d0f25c00 x1876699504413568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763864.254567:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763864.254605:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504413952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763864.254607:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504413568/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763864.254656:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763864.254660:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763864.254663:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763864.254670:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763864.254674:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763864.254682:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3790 00000100:00100000:1.0:1789763864.254689:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504413568:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763864.254693:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763864.254708:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763864.254716:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763864.254766:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504414080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763864.254772:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504413696:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763864.254789:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763864.254796:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763864.254798:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763864.254804:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3790 00000100:00080000:2.0:1789763864.254822:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763864.254827:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763864.254835:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763864.254844:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763864.254855:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763864/real 1789763864] req@ffff9018d0f26a00 x1876699504413696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763864.254867:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504414208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763864.254893:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504413696/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763864.254908:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763864.254914:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763864.254921:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504413696:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763864.254930:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504413824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763864.254954:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763864.254966:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763864/real 1789763864] req@ffff9018d0f26680 x1876699504413824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763864.254981:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504413824/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763864.255001:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763864.255005:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763864.255010:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504413824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763864.255016:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504413952:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763864.255080:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763864.255095:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763864/real 1789763864] req@ffff9018d0f24000 x1876699504413952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763864.255110:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504413952/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763864.255121:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763864.255125:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763864.255129:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504413952:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763864.255152:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504414080:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763864.255176:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763864.255188:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763864/real 1789763864] req@ffff9018d0f26300 x1876699504414080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763864.255203:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504414080/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763864.255213:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763864.255217:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763864.255222:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504414080:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763864.255229:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504414208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763864.255249:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763864.255259:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763864/real 1789763864] req@ffff9018d0f24380 x1876699504414208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763864.255273:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504414208/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763864.255282:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763864.255286:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763864.255289:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504414208:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763864.894257:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763865.918173:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763866.942218:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763867.966268:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763868.990248:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763869.374189:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763869.374205:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763869.374212:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763869.374226:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3795 00000100:00080000:2.0:1789763869.374242:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763869.374252:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763869.374313:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504414336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763869.374355:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763869.374362:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763869.374366:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763869.374380:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3795 00000100:00080000:2.0:1789763869.374387:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763869.374397:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763869.374413:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763869.374426:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504414336:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763869.374439:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504414464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763869.374481:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763869.374488:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763869.374492:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763869.374501:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3795 00000100:00080000:2.0:1789763869.374510:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763869.374518:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763869.374529:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763869.374536:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763869.374563:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504414592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763869.374592:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763869/real 1789763869] req@ffff9018d0f26d80 x1876699504414336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763869.374603:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763869.374618:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763869.374623:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763869.374628:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504414336/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763869.374638:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3795 00000100:00080000:1.0:1789763869.374653:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763869.374656:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763869.374663:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763869.374677:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763869.374678:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504414336:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789763869.374724:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763869.374756:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504414464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763869.374769:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504414720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763869.374796:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789763869.374801:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763869.374813:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763869.374819:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763869.374831:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763869/real 1789763869] req@ffff9018d0f25180 x1876699504414464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763869.374832:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3795 00000100:00080000:2.0:1789763869.374849:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763869.374864:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504414464/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763869.374867:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763869.374887:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763869.374889:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763869.374902:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763869.374908:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504414464:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763869.374924:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504414592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763869.374928:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504414848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763869.374952:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789763869.374955:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763869.374967:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763869.374973:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763869.374976:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763869/real 1789763869] req@ffff9018d0f27100 x1876699504414592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763869.374991:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3795 00000100:00080000:2.0:1789763869.375006:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763869.375007:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504414592/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763869.375023:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763869.375059:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763869.375067:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763869.375072:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504414592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763869.375093:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504414720:192.168.203.158@tcp:8:kworker.0 00000008:00100000:2.0:1789763869.375102:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763869.375127:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763869.375144:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504414976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763869.375151:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763869/real 1789763869] req@ffff9018d0f25f80 x1876699504414720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763869.375212:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504414720/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763869.375226:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763869.375231:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763869.375236:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504414720:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763869.375246:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504414848:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763869.375268:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763869.375280:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763869/real 1789763869] req@ffff9018d0f26d80 x1876699504414848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763869.375303:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504414848/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763869.375313:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763869.375316:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763869.375320:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504414848:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763869.375336:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504414976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763869.375355:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763869.375366:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763869/real 1789763869] req@ffff9018d0f24380 x1876699504414976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763869.375380:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504414976/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763869.375389:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763869.375393:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763869.375398:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504414976:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763870.014269:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763871.038205:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763872.062286:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763873.086136:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763874.110140:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763874.494142:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763874.494151:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763874.494155:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763874.494165:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3801 00000100:00080000:0.0:1789763874.494175:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763874.494182:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763874.494219:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504415104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763874.494257:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763874.494270:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763874.494273:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763874.494279:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3801 00000100:00080000:0.0:1789763874.494285:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763874.494291:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763874.494304:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763874.494321:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504415104:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763874.494326:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504415232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763874.494346:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763874.494351:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763874.494354:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763874.494360:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3801 00000100:00080000:0.0:1789763874.494366:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763874.494370:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763874.494375:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763874.494389:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504415360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763874.494399:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763874.494407:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763874.494412:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763874.494415:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763874.494420:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3801 00000100:00080000:0.0:1789763874.494423:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763874.494426:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763874.494432:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763874.494447:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763874/real 1789763874] req@ffff9018c8b99500 x1876699504415104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763874.494448:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504415488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763874.494489:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763874.494492:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763874.494496:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763874.494497:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504415104/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763874.494503:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3801 00000100:00080000:0.0:1789763874.494509:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763874.494516:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763874.494519:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789763874.494521:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763874.494527:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763874.494535:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504415616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763874.494536:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504415104:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763874.494553:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763874.494556:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763874.494558:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763874.494562:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3801 00000100:00080000:0.0:1789763874.494565:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763874.494570:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504415232:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763874.494571:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763874.494580:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763874.494594:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504415744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763874.494604:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763874.494620:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763874/real 1789763874] req@ffff9018c8b98380 x1876699504415232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763874.494633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504415232/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763874.494641:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763874.494644:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763874.494666:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504415232:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763874.494672:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504415360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763874.494710:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763874.494723:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763874/real 1789763874] req@ffff9018c8b9a300 x1876699504415360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763874.494735:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504415360/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763874.494744:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763874.494747:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763874.494751:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504415360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763874.494756:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504415488:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763874.494772:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763874.494781:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763874/real 1789763874] req@ffff9018c8b9ad80 x1876699504415488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763874.494793:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504415488/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763874.494801:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763874.494821:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763874.494824:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504415488:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763874.494832:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504415616:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763874.494852:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763874.494862:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763874/real 1789763874] req@ffff9018c8b99c00 x1876699504415616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763874.494893:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504415616/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763874.494903:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763874.494906:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763874.494909:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504415616:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763874.494927:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504415744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763874.494962:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763874.494974:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763874/real 1789763874] req@ffff9018c8b99500 x1876699504415744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763874.494991:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504415744/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763874.495004:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763874.495008:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763874.495014:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504415744:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763875.134185:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763876.158325:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763877.182306:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763878.206317:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763879.230287:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763879.614197:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763879.614207:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763879.614212:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763879.614223:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3806 00000100:00080000:0.0:1789763879.614236:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763879.614243:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763879.614285:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504415872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763879.614319:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763879.614323:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763879.614325:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763879.614329:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3806 00000100:00080000:0.0:1789763879.614332:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763879.614336:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763879.614346:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763879.614359:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504416000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763879.614376:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763879.614381:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763879.614384:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763879.614389:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3806 00000100:00080000:0.0:1789763879.614395:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763879.614402:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763879.614410:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763879.614429:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504416128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763879.614443:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504415872:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763879.614470:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763879.614475:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763879.614477:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763879.614483:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3806 00000100:00080000:0.0:1789763879.614489:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763879.614494:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763879.614501:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763879.614516:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504416256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763879.614527:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789763879.614535:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763879.614536:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763879.614542:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763879.614547:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3806 00000100:00080000:0.0:1789763879.614552:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763879.614557:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763879.614561:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763879.614573:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504416384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763879.614582:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763879.614586:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763879.614588:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763879.614593:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3806 00000100:00080000:0.0:1789763879.614598:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763879.614607:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763879.614607:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763879/real 1789763879] req@ffff9018c8b99880 x1876699504415872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763879.614637:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763879.614682:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504416512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763879.614698:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504415872/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.614724:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.614733:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763879.614743:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504415872:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763879.614758:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504416000:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763879.614800:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763879.614823:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763879/real 1789763879] req@ffff9018c8b98000 x1876699504416000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.614840:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504416000/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.614850:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.614854:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763879.614858:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504416000:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763879.614886:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504416128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763879.614910:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763879.614921:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763879/real 1789763879] req@ffff9018c8b9b800 x1876699504416128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.614934:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504416128/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.614944:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.614948:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763879.614951:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504416128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763879.614957:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504416256:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763879.614991:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763879.615001:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763879/real 1789763879] req@ffff9018c8b9aa00 x1876699504416256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.615012:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504416256/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.615021:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.615024:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763879.615027:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504416256:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763879.615032:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504416384:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763879.615047:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763879.615055:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763879/real 1789763879] req@ffff9018c8b98e00 x1876699504416384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.615099:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504416384/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.615108:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.615111:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763879.615114:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504416384:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763879.615120:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504416512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763879.615138:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763879.615149:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763879/real 1789763879] req@ffff9018c8b9a680 x1876699504416512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.615161:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504416512/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.615169:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.615172:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763879.615176:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504416512:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763880.254236:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763881.278299:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763882.302305:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763883.326179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763884.350290:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763884.734187:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763884.734200:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763884.734206:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763884.734218:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3811 00000100:00080000:0.0:1789763884.734233:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763884.734242:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763884.734291:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504416640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763884.734327:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763884.734332:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763884.734335:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763884.734340:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3811 00000100:00080000:0.0:1789763884.734344:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763884.734349:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763884.734361:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763884.734376:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504416768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763884.734390:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763884.734395:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763884.734398:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763884.734402:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3811 00000100:00080000:0.0:1789763884.734406:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763884.734415:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504416640:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763884.734417:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763884.734432:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763884.734451:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504416896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763884.734465:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763884.734469:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763884.734472:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763884.734477:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3811 00000100:00080000:0.0:1789763884.734496:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763884.734501:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763884.734508:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763884.734514:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763884.734534:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504417024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763884.734557:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763884.734565:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763884.734568:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763884.734601:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3811 00000100:00100000:1.0:1789763884.734611:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763884/real 1789763884] req@ffff9018c8b98a80 x1876699504416640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763884.734615:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763884.734633:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763884.734649:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763884.734651:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504416640/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763884.734674:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763884.734686:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763884.734690:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504417152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763884.734694:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504416640:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763884.734718:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504416768:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763884.734725:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763884.734732:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763884.734735:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763884.734743:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3811 00000100:00080000:0.0:1789763884.734755:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763884.734765:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763884.734766:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763884.734779:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763884.734794:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763884/real 1789763884] req@ffff9018c8b98700 x1876699504416768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763884.734803:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504417280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763884.734837:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504416768/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763884.734850:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763884.734856:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763884.734861:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504416768:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763884.734896:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504416896:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763884.734926:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763884.734943:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763884/real 1789763884] req@ffff9018c8b9bb80 x1876699504416896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763884.734959:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504416896/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763884.734971:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763884.734976:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763884.734982:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504416896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763884.734992:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504417024:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763884.735014:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763884.735027:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763884/real 1789763884] req@ffff9018c8b9b100 x1876699504417024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763884.735084:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504417024/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763884.735095:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763884.735099:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763884.735103:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504417024:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763884.735111:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504417152:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763884.735135:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763884.735148:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763884/real 1789763884] req@ffff9018c8b9b480 x1876699504417152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763884.735163:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504417152/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763884.735173:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763884.735176:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763884.735180:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504417152:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763884.735187:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504417280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763884.735205:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763884.735217:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763884/real 1789763884] req@ffff9018c8b99f80 x1876699504417280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763884.735240:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504417280/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763884.735255:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763884.735262:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763884.735268:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504417280:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763885.374172:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763886.398221:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763887.422282:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763888.446211:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763889.470194:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763889.854225:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763889.854243:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763889.854250:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763889.854267:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3816 00000100:00080000:3.0:1789763889.854286:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763889.854298:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763889.854349:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d880 x1876699504417408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763889.854384:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763889.854389:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763889.854392:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763889.854397:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3816 00000100:00080000:3.0:1789763889.854404:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763889.854409:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763889.854421:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763889.854437:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504417408:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763889.854441:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070fb80 x1876699504417536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763889.854464:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763889.854470:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763889.854473:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763889.854479:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3816 00000100:00080000:3.0:1789763889.854483:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763889.854488:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763889.854494:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763889.854509:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f480 x1876699504417664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763889.854522:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763889.854527:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763889.854529:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763889.854564:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3816 00000100:00080000:3.0:1789763889.854575:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763889.854583:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763889.854594:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763889.854615:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504417536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:3.0:1789763889.854629:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ed80 x1876699504417792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763889.854658:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763889.854665:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789763889.854669:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504417664:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789763889.854673:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763889.854689:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3816 00000100:00080000:3.0:1789763889.854697:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763889.854704:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763889.854715:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763889.854724:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504417792:192.168.203.158@tcp:8:kworker.0 00000100:00100000:3.0:1789763889.854742:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ce00 x1876699504417920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763889.854761:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763889.854766:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763889.854770:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763889.854772:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504417920:192.168.203.158@tcp:38:kworker.0 00000100:00080000:3.0:1789763889.854781:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3816 00000100:00080000:3.0:1789763889.854789:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763889.854794:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763889.854801:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763889.854823:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d180 x1876699504418048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763889.854896:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504418048:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763890.494258:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763891.518241:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763892.542235:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763892.926465:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:0.0:1789763892.926559:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763892.926587:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763892.926670:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763892.926700:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763892.926725:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763892.926748:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763892.926775:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789763892.926781:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763889/real 1789763892] req@ffff9018d070ed80 x1876699504417792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763892.926832:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ed80 x1876699504417792/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763892.926859:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763892.926870:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763892.926879:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504417792:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763892.926894:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763889/real 1789763892] req@ffff9018d070fb80 x1876699504417536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763892.926923:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070fb80 x1876699504417536/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763892.926940:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763892.926948:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763892.926955:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504417536:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763892.926969:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763889/real 1789763892] req@ffff9018d070f480 x1876699504417664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763892.926993:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f480 x1876699504417664/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763892.927009:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763892.927018:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763892.927024:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504417664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763892.927089:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763889/real 1789763892] req@ffff9018d070d880 x1876699504417408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763892.927119:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d880 x1876699504417408/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763892.927134:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763892.927142:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763892.927149:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504417408:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763892.927205:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763889/real 1789763892] req@ffff9018d070d180 x1876699504418048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763892.927233:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d180 x1876699504418048/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763892.927250:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763892.927258:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763892.927265:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504418048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763892.927279:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763889/real 1789763892] req@ffff9018d070ce00 x1876699504417920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763892.927304:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ce00 x1876699504417920/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763892.927320:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763892.927329:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763892.927335:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504417920:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763893.566203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763894.590281:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763894.974292:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763894.974303:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763894.974309:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763894.974324:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3821 00000100:00080000:0.0:1789763894.974339:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763894.974350:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763894.974398:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504418176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763894.974437:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763894.974443:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763894.974447:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763894.974453:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3821 00000100:00080000:0.0:1789763894.974460:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763894.974466:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763894.974480:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763894.974530:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504418304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763894.974555:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763894.974563:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763894.974568:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763894.974577:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3821 00000100:00080000:0.0:1789763894.974585:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763894.974595:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763894.974607:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763894.974636:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504418432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763894.974644:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504418176:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763894.974677:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763894.974685:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763894.974689:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763894.974699:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3821 00000100:00080000:0.0:1789763894.974708:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763894.974718:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763894.974730:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789763894.974768:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763894.974778:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504418560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763894.974806:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763894.974814:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763894.974818:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763894.974828:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3821 00000100:00080000:0.0:1789763894.974838:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763894.974854:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763894.974855:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763894/real 1789763894] req@ffff9018c8b99f80 x1876699504418176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763894.974869:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763894.974903:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504418688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763894.974917:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504418176/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763894.974930:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763894.974948:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763894.974954:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763894.974964:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763894.974970:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3821 00000100:00080000:2.0:1789763894.974982:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789763894.974984:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763894.974999:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504418176:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763894.975002:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763894.975024:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763894.975077:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504418304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763894.975110:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504418816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763894.975142:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763894.975178:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763894/real 1789763894] req@ffff9018c8b9b480 x1876699504418304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763894.975207:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504418304/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763894.975228:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763894.975239:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763894.975247:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504418304:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763894.975300:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504418432:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763894.975348:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763894.975374:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763894/real 1789763894] req@ffff9018c8b9b100 x1876699504418432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763894.975410:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504418432/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763894.975428:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763894.975435:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763894.975442:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504418432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763894.975456:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504418560:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763894.975531:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763894.975558:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763894/real 1789763894] req@ffff9018c8b9bb80 x1876699504418560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763894.975584:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504418560/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763894.975602:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763894.975608:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763894.975615:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504418560:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763894.975628:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504418688:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763894.975669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763894.975694:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763894/real 1789763894] req@ffff9018c8b98700 x1876699504418688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763894.975720:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504418688/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763894.975737:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763894.975743:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763894.975749:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504418688:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763894.975763:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504418816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763894.975805:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763894.975829:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763894/real 1789763894] req@ffff9018c8b98a80 x1876699504418816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763894.975853:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504418816/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763894.975870:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763894.975877:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763894.975883:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504418816:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763895.614234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763896.638276:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763897.662253:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763898.686237:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763899.710254:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763900.094143:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763900.094152:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763900.094157:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763900.094167:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3826 00000100:00080000:2.0:1789763900.094177:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763900.094184:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763900.094222:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504418944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763900.094249:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763900.094253:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763900.094255:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763900.094258:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3826 00000100:00080000:2.0:1789763900.094261:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763900.094283:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763900.094296:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763900.094315:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504419072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763900.094333:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504418944:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763900.094336:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763900.094349:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763900.094352:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763900.094358:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3826 00000100:00080000:2.0:1789763900.094362:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763900.094366:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763900.094372:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763900.094389:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504419200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763900.094401:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763900.094405:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763900.094408:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763900.094412:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3826 00000100:00080000:2.0:1789763900.094429:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763900.094434:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763900.094441:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763900.094446:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763900.094500:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504419328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763900.094522:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763900.094526:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763900.094527:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763900.094532:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3826 00000100:00080000:2.0:1789763900.094537:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763900.094542:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763900.094547:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763900.094570:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763900/real 1789763900] req@ffff9018d0f26300 x1876699504418944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763900.094588:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504419456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763900.094604:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763900.094612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504418944/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763900.094613:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763900.094620:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763900.094634:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3826 00000100:00080000:2.0:1789763900.094644:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763900.094651:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763900.094656:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763900.094663:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789763900.094670:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504419584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763900.094686:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763900.094704:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504418944:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763900.094738:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504419072:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763900.094782:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763900.094808:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763900/real 1789763900] req@ffff9018d0f24000 x1876699504419072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763900.094836:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504419072/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763900.094870:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763900.094875:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763900.094880:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504419072:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763900.094887:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504419200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763900.094915:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763900.094932:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763900/real 1789763900] req@ffff9018d0f26680 x1876699504419200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763900.094960:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504419200/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763900.094971:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763900.094976:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763900.094980:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504419200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763900.094989:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504419328:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763900.095014:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763900.095025:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763900/real 1789763900] req@ffff9018d0f26a00 x1876699504419328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763900.095036:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504419328/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763900.095044:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763900.095047:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763900.095075:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504419328:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763900.095080:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504419456:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763900.095099:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763900.095108:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763900/real 1789763900] req@ffff9018d0f25c00 x1876699504419456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763900.095125:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504419456/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763900.095132:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763900.095135:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763900.095137:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504419456:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763900.095142:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504419584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763900.095155:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763900.095184:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763900/real 1789763900] req@ffff9018d0f27800 x1876699504419584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763900.095199:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504419584/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763900.095208:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763900.095212:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763900.095215:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504419584:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763900.734246:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763901.758259:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763902.782245:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763903.806253:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763904.830244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763905.214257:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763905.214270:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763905.214275:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763905.214287:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3831 00000100:00080000:0.0:1789763905.214300:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763905.214310:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763905.214363:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504419712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763905.214411:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763905.214419:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763905.214445:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763905.214455:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3831 00000100:00080000:0.0:1789763905.214464:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763905.214473:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763905.214491:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763905.214513:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504419712:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763905.214521:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504419840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763905.214544:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763905.214549:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763905.214551:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763905.214557:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3831 00000100:00080000:0.0:1789763905.214565:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763905.214570:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763905.214577:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763905.214595:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504419968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763905.214607:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763905.214612:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763905.214614:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763905.214619:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3831 00000100:00080000:0.0:1789763905.214624:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763905.214631:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763905.214638:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763905.214647:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763905.214663:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504420096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763905.214677:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763905.214682:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763905.214685:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763905.214691:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3831 00000100:00100000:1.0:1789763905.214698:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763905/real 1789763905] req@ffff9018c8b9a680 x1876699504419712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763905.214699:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763905.214712:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763905.214724:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763905.214745:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504420224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763905.214751:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504419712/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763905.214820:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763905.214823:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763905.214832:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763905.214838:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789763905.214839:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763905.214852:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3831 00000100:00100000:1.0:1789763905.214860:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504419712:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763905.214861:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763905.214875:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763905.214883:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763905.214898:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504420352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763905.214917:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504419840:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763905.214959:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763905.214978:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763905/real 1789763905] req@ffff9018c8b98e00 x1876699504419840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763905.214995:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504419840/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763905.215006:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763905.215011:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763905.215016:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504419840:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763905.215087:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504419968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763905.215134:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763905.215157:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763905/real 1789763905] req@ffff9018c8b9aa00 x1876699504419968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763905.215174:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504419968/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763905.215185:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763905.215190:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763905.215194:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504419968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763905.215203:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504420096:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763905.215229:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763905.215242:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763905/real 1789763905] req@ffff9018c8b9b800 x1876699504420096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763905.215257:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504420096/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763905.215267:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763905.215271:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763905.215274:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504420096:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763905.215281:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504420224:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763905.215301:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763905.215312:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763905/real 1789763905] req@ffff9018c8b98000 x1876699504420224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763905.215326:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504420224/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763905.215336:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763905.215339:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763905.215342:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504420224:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763905.215365:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504420352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763905.215385:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763905.215397:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763905/real 1789763905] req@ffff9018c8b99880 x1876699504420352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763905.215412:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504420352/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763905.215452:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763905.215459:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763905.215464:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504420352:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763905.854262:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763906.878309:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763907.902135:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763908.926312:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763909.950251:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763910.334206:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763910.334220:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763910.334226:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763910.334241:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3836 00000100:00080000:2.0:1789763910.334257:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763910.334269:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763910.334319:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504420480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763910.334357:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763910.334362:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763910.334364:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763910.334369:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3836 00000100:00080000:2.0:1789763910.334374:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763910.334379:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763910.334417:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763910.334438:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504420480:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763910.334443:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504420608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763910.334473:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763910.334480:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763910.334484:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763910.334493:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3836 00000100:00080000:2.0:1789763910.334502:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763910.334511:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763910.334522:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763910.334552:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504420736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763910.334554:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763910.334573:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763910.334579:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763910.334582:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763910.334589:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3836 00000100:00080000:2.0:1789763910.334597:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763910.334602:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763910.334610:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763910.334632:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504420864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763910.334638:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763910/real 1789763910] req@ffff9018d0f24700 x1876699504420480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763910.334684:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763910.334695:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504420480/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763910.334718:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763910.334724:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763910.334725:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763910.334740:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763910.334743:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3836 00000100:00080000:2.0:1789763910.334755:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763910.334756:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504420480:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763910.334770:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763910.334779:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763910.334800:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504420992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763910.334804:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504420608:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763910.334832:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763910.334839:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763910.334844:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763910.334854:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3836 00000800:00000100:1.0:1789763910.334861:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763910.334884:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763910.334898:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763910.334907:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763910/real 1789763910] req@ffff9018d0f24e00 x1876699504420608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763910.334915:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763910.334948:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504421120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763910.334956:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504420608/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763910.335008:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763910.335014:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763910.335025:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504420608:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763910.335092:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504420736:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763910.335138:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763910.335156:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763910/real 1789763910] req@ffff9018d0f27480 x1876699504420736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763910.335173:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504420736/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763910.335184:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763910.335188:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763910.335192:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504420736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763910.335200:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504420864:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763910.335223:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763910.335235:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763910/real 1789763910] req@ffff9018d0f25500 x1876699504420864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763910.335251:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504420864/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763910.335260:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763910.335264:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763910.335269:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504420864:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763910.335291:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504420992:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763910.335311:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763910.335323:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763910/real 1789763910] req@ffff9018d0f24a80 x1876699504420992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763910.335338:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504420992/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763910.335348:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763910.335351:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763910.335355:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504420992:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763910.335362:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504421120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763910.335383:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763910.335431:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763910/real 1789763910] req@ffff9018d0f24700 x1876699504421120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763910.335457:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504421120/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763910.335474:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763910.335481:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763910.335488:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504421120:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763910.974234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763911.998325:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763913.022307:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763914.046246:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763915.070289:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763915.454249:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763915.454264:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763915.454271:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763915.454287:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3842 00000100:00080000:1.0:1789763915.454304:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763915.454315:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763915.454413:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504421248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763915.454461:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763915.454466:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763915.454469:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763915.454475:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3842 00000100:00080000:1.0:1789763915.454479:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763915.454485:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763915.454500:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763915.454519:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504421376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763915.454540:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763915.454541:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504421248:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763915.454564:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763915.454569:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763915.454575:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3842 00000100:00080000:1.0:1789763915.454580:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763915.454587:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763915.454597:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763915.454613:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504421504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763915.454634:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763915.454640:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763915.454643:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763915.454648:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3842 00000100:00080000:1.0:1789763915.454652:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763915.454661:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789763915.454674:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:1.0:1789763915.454677:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763915.454695:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504421632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763915.454709:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763915.454714:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763915.454716:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763915.454721:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3842 00000100:00080000:1.0:1789763915.454727:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763915.454734:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763915.454740:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763915.454753:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504421760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763915.454762:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763915/real 1789763915] req@ffff9018c66a6300 x1876699504421248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763915.454775:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763915.454800:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763915.454810:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763915.454823:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3842 00000100:00080000:1.0:1789763915.454829:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763915.454835:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504421248/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763915.454845:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763915.454864:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763915.454881:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789763915.454895:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504421888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763915.454901:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763915.454924:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504421248:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763915.454987:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504421376:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763915.455026:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763915.455081:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763915/real 1789763915] req@ffff9018c66a5180 x1876699504421376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763915.455098:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504421376/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763915.455110:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763915.455114:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763915.455119:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504421376:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763915.455128:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504421504:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763915.455153:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763915.455165:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763915/real 1789763915] req@ffff9018c66a5500 x1876699504421504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763915.455180:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504421504/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763915.455189:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763915.455193:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763915.455197:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504421504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763915.455204:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504421632:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763915.455224:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763915.455236:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763915/real 1789763915] req@ffff9018c66a6680 x1876699504421632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763915.455250:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504421632/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763915.455260:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763915.455263:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763915.455267:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504421632:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763915.455274:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504421760:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763915.455292:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763915.455303:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763915/real 1789763915] req@ffff9018c66a4700 x1876699504421760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763915.455317:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504421760/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763915.455327:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763915.455330:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763915.455334:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504421760:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763915.455340:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504421888:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763915.455394:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763915.455410:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763915/real 1789763915] req@ffff9018c66a4e00 x1876699504421888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763915.455423:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504421888/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763915.455433:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763915.455437:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763915.455441:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504421888:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763916.094316:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763917.118309:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763918.142296:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763919.166153:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763920.190180:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763920.574114:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763920.574122:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763920.574126:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763920.574134:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3847 00000100:00080000:0.0:1789763920.574143:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763920.574149:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763920.574182:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504422016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763920.574204:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763920.574207:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763920.574208:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763920.574211:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3847 00000100:00080000:0.0:1789763920.574213:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763920.574216:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763920.574224:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763920.574233:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504422144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763920.574241:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763920.574244:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763920.574246:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763920.574248:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504422016:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763920.574250:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3847 00000100:00080000:0.0:1789763920.574256:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763920.574259:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763920.574262:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763920.574271:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504422272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763920.574278:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763920.574281:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763920.574282:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763920.574285:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3847 00000100:00080000:0.0:1789763920.574287:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763920.574290:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763920.574293:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789763920.574322:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763920.574326:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504422400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763920.574335:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763920.574338:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763920.574340:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763920.574343:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3847 00000100:00080000:0.0:1789763920.574346:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763920.574349:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763920.574354:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763920.574363:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763920/real 1789763920] req@ffff9018c8b99500 x1876699504422016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763920.574366:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504422528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763920.574394:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763920.574401:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763920.574403:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763920.574409:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3847 00000100:00080000:0.0:1789763920.574412:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763920.574414:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763920.574418:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504422016/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763920.574419:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763920.574432:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789763920.574433:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504422656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763920.574439:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763920.574447:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504422016:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763920.574470:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504422144:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763920.574487:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763920.574495:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763920/real 1789763920] req@ffff9018c8b99c00 x1876699504422144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763920.574503:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504422144/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763920.574509:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763920.574511:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763920.574513:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504422144:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763920.574517:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504422272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763920.574527:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763920.574532:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763920/real 1789763920] req@ffff9018c8b9ad80 x1876699504422272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763920.574540:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504422272/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763920.574549:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763920.574551:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763920.574553:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504422272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763920.574556:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504422400:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763920.574566:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763920.574572:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763920/real 1789763920] req@ffff9018c8b9a300 x1876699504422400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763920.574579:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504422400/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763920.574584:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763920.574586:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763920.574588:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504422400:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763920.574591:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504422528:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763920.574600:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763920.574606:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763920/real 1789763920] req@ffff9018c8b98380 x1876699504422528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763920.574612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504422528/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763920.574617:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763920.574619:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763920.574621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504422528:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763920.574625:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504422656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763920.574635:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763920.574640:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763920/real 1789763920] req@ffff9018c8b99880 x1876699504422656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763920.574647:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504422656/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763920.574652:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763920.574653:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763920.574655:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504422656:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763921.214204:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763922.238230:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763923.262215:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763924.286364:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763925.310194:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763925.694172:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763925.694185:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763925.694191:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763925.694204:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3852 00000100:00080000:0.0:1789763925.694217:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763925.694226:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763925.694302:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504422784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763925.694351:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763925.694357:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763925.694368:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763925.694375:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3852 00000100:00080000:0.0:1789763925.694380:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763925.694387:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763925.694401:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763925.694422:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504422912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763925.694438:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763925.694443:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763925.694445:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763925.694454:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3852 00000100:00080000:0.0:1789763925.694460:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763925.694473:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763925.694473:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504422784:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789763925.694492:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763925.694520:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504423040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763925.694540:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763925.694548:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763925.694552:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763925.694562:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3852 00000100:00080000:0.0:1789763925.694574:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763925.694583:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763925.694595:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763925.694615:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504423168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763925.694627:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763925.694646:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763925.694651:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763925.694656:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763925.694667:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3852 00000100:00080000:0.0:1789763925.694674:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763925.694682:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763925.694689:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763925.694708:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504423296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763925.694720:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763925/real 1789763925] req@ffff9018c8b98000 x1876699504422784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763925.694750:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763925.694776:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763925.694780:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763925.694795:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3852 00000100:00080000:0.0:1789763925.694801:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763925.694807:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763925.694815:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763925.694829:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504422784/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763925.694830:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504423424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763925.694887:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763925.694911:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763925.694942:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504422784:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763925.695004:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504422912:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763925.695134:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763925.695156:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763925/real 1789763925] req@ffff9018c8b9b800 x1876699504422912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763925.695175:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504422912/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763925.695187:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763925.695192:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763925.695196:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504422912:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763925.695206:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504423040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763925.695244:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763925.695267:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763925/real 1789763925] req@ffff9018c8b9aa00 x1876699504423040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763925.695324:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504423040/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763925.695342:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763925.695347:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763925.695352:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504423040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763925.695361:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504423168:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763925.695396:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763925.695418:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763925/real 1789763925] req@ffff9018c8b98e00 x1876699504423168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763925.695456:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504423168/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763925.695475:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763925.695483:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763925.695490:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504423168:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763925.695503:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504423296:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763925.695536:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763925.695552:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763925/real 1789763925] req@ffff9018c8b9a680 x1876699504423296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763925.695567:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504423296/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763925.695578:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763925.695582:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763925.695586:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504423296:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763925.695594:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504423424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763925.695614:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763925.695625:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763925/real 1789763925] req@ffff9018c8b98a80 x1876699504423424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763925.695639:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504423424/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763925.695657:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763925.695662:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763925.695666:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504423424:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763926.334167:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763927.358243:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763928.382204:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763929.406303:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763930.430323:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763930.814277:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763930.814292:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763930.814300:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763930.814316:0:11323:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3857 00000100:00080000:0.0:1789763930.814330:0:11323:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763930.814340:0:11323:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763930.814385:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504423552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763930.814423:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763930.814429:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763930.814433:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763930.814440:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3857 00000100:00080000:0.0:1789763930.814446:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763930.814453:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763930.814463:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504423552:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789763930.814472:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763930.814509:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504423680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763930.814528:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763930.814533:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763930.814536:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763930.814544:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3857 00000100:00080000:0.0:1789763930.814551:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763930.814560:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763930.814561:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763930.814574:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763930.814599:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504423808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763930.814621:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763930.814624:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763930/real 1789763930] req@ffff9018c8b98700 x1876699504423552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763930.814635:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763930.814646:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763930.814664:0:11323:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3857 00000100:00100000:2.0:1789763930.814670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504423552/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763930.814677:0:11323:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763930.814691:0:11323:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763930.814699:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1789763930.814715:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763930.814726:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763930.814732:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504423552:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763930.814762:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504423680:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763930.814791:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763930.814820:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763930/real 1789763930] req@ffff9018c8b9bb80 x1876699504423680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763930.814822:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504423936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763930.814876:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504423680/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763930.814885:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763930.814894:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763930.814902:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789763930.814908:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789763930.814909:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504423680:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763930.814920:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763930.814923:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504423808:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763930.814942:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3857 00000100:00080000:0.0:1789763930.814955:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763930.814962:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763930.814973:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763930.814982:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763930/real 1789763930] req@ffff9018c8b9b100 x1876699504423808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763930.814986:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763930.815013:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504423808/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763930.815023:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504424064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763930.815075:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763930.815086:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763930.815099:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504423808:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763930.815105:0:11323:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763930.815113:0:11323:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763930.815117:0:11323:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763930.815126:0:11323:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3857 00000100:00080000:0.0:1789763930.815133:0:11323:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763930.815135:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504423936:192.168.203.158@tcp:8:kworker.0 00000100:00080000:0.0:1789763930.815145:0:11323:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763930.815157:0:11323:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763930.815179:0:11323:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504424192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763930.815195:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763930.815268:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763930/real 1789763930] req@ffff9018c8b9b480 x1876699504423936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763930.815304:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504423936/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763930.815322:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763930.815328:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763930.815334:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504423936:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763930.815344:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504424064:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763930.815379:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763930.815402:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763930/real 1789763930] req@ffff9018c8b99f80 x1876699504424064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763930.815420:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504424064/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763930.815433:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763930.815438:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763930.815442:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504424064:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763930.815461:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504424192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763930.815482:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763930.815493:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763930/real 1789763930] req@ffff9018c8b9b100 x1876699504424192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763930.815507:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504424192/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763930.815517:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763930.815520:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763930.815523:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504424192:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763931.454327:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763932.478388:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763933.502259:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763934.526317:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763935.550281:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763935.934150:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763935.934164:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763935.934170:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763935.934184:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3862 00000100:00080000:2.0:1789763935.934219:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763935.934233:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763935.934279:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504424320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763935.934330:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763935.934338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763935.934342:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763935.934351:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3862 00000100:00080000:2.0:1789763935.934361:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763935.934371:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763935.934389:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763935.934392:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504424320:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763935.934416:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504424448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763935.934435:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763935.934440:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763935.934443:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763935.934450:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3862 00000100:00080000:2.0:1789763935.934456:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763935.934461:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763935.934469:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763935.934483:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763935.934490:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504424576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763935.934520:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763935.934527:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763935.934531:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763935.934543:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763935/real 1789763935] req@ffff9018d0f27800 x1876699504424320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763935.934544:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3862 00000100:00080000:2.0:1789763935.934559:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763935.934575:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763935.934584:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504424320/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763935.934605:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1789763935.934608:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763935.934618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763935.934625:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504424320:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763935.934634:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504424704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763935.934651:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763935.934656:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763935.934660:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763935.934667:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504424448:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763935.934676:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3862 00000100:00080000:2.0:1789763935.934689:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763935.934699:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:0.0:1789763935.934711:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:2.0:1789763935.934713:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763935.934735:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763935/real 1789763935] req@ffff9018d0f25c00 x1876699504424448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763935.934747:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504424832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763935.934763:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504424448/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763935.934803:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763935.934806:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763935.934815:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763935.934821:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763935.934824:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763935.934834:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3862 00000100:00080000:2.0:1789763935.934841:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763935.934847:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763935.934855:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504424448:192.168.203.158@tcp:38:kworker.0 00000008:00100000:2.0:1789763935.934859:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763935.934885:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504424576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763935.934898:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504424960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763935.934946:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763935.934988:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763935/real 1789763935] req@ffff9018d0f26a00 x1876699504424576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763935.935004:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504424576/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763935.935015:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763935.935019:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763935.935023:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504424576:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763935.935031:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504424704:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763935.935092:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763935.935108:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763935/real 1789763935] req@ffff9018d0f26680 x1876699504424704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763935.935123:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504424704/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763935.935133:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763935.935137:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763935.935142:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504424704:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763935.935166:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504424832:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763935.935187:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763935.935247:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763935/real 1789763935] req@ffff9018d0f24000 x1876699504424832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763935.935271:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504424832/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763935.935284:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763935.935288:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763935.935292:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504424832:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763935.935301:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504424960:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763935.935330:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763935.935344:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763935/real 1789763935] req@ffff9018d0f26300 x1876699504424960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763935.935358:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504424960/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763935.935368:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763935.935371:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763935.935374:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504424960:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763936.574245:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763937.598357:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763938.622217:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763939.646211:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763940.670296:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763941.054227:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763941.054244:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763941.054251:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763941.054267:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3867 00000100:00080000:1.0:1789763941.054284:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763941.054297:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763941.054350:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504425088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763941.054390:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763941.054396:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763941.054399:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763941.054404:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3867 00000100:00080000:1.0:1789763941.054410:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763941.054415:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763941.054428:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763941.054447:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504425216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763941.054463:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763941.054472:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504425088:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763941.054473:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763941.054478:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763941.054491:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3867 00000100:00080000:1.0:1789763941.054499:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763941.054507:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763941.054517:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763941.054544:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504425344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763941.054566:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763941.054575:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763941.054578:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763941.054591:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763941.054592:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3867 00000100:00080000:1.0:1789763941.054608:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763941.054615:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763941.054625:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763941.054648:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504425472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763941.054660:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763941/real 1789763941] req@ffff9018c66a7800 x1876699504425088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763941.054669:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763941.054685:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763941.054692:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763941.054702:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3867 00000100:00080000:1.0:1789763941.054707:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763941.054712:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504425088/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763941.054720:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763941.054733:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789763941.054744:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763941.054757:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763941.054757:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504425600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763941.054772:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504425088:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763941.054792:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763941.054798:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763941.054800:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763941.054805:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3867 00000100:00080000:1.0:1789763941.054808:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763941.054813:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763941.054835:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763941.054838:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504425216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763941.054854:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504425728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763941.054887:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763941.054913:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763941/real 1789763941] req@ffff9018c66a7b80 x1876699504425216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763941.054930:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504425216/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763941.054941:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763941.054945:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763941.054950:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504425216:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763941.054957:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504425344:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763941.054978:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763941.054989:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763941/real 1789763941] req@ffff9018c66a4000 x1876699504425344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763941.055003:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504425344/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763941.055022:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763941.055026:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763941.055030:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504425344:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763941.055081:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504425472:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763941.055109:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763941.055123:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763941/real 1789763941] req@ffff9018c66a5f80 x1876699504425472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763941.055138:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504425472/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763941.055148:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763941.055152:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763941.055156:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504425472:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763941.055194:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504425600:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763941.055226:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763941.055242:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763941/real 1789763941] req@ffff9018c66a6d80 x1876699504425600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763941.055257:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504425600/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763941.055268:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763941.055272:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763941.055276:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504425600:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763941.055303:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504425728:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763941.055323:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763941.055335:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763941/real 1789763941] req@ffff9018c66a5c00 x1876699504425728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763941.055349:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504425728/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763941.055358:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763941.055362:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763941.055365:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504425728:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763941.694290:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763942.718286:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763943.742303:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763944.766317:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763945.790338:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763946.174213:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763946.174231:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763946.174238:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763946.174254:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3872 00000100:00080000:2.0:1789763946.174272:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763946.174284:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763946.174334:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504425856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763946.174371:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763946.174376:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763946.174379:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763946.174385:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3872 00000100:00080000:2.0:1789763946.174389:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763946.174394:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763946.174406:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763946.174422:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504425984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763946.174454:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763946.174459:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763946.174463:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763946.174469:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504425856:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763946.174475:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3872 00000100:00080000:2.0:1789763946.174489:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763946.174497:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763946.174509:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763946.174539:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504426112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763946.174561:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789763946.174571:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763946.174580:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763946.174587:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763946.174599:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3872 00000100:00080000:2.0:1789763946.174606:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763946.174613:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763946.174624:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763946.174633:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763946/real 1789763946] req@ffff9018d0f24380 x1876699504425856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763946.174648:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504426240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763946.174668:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763946.174670:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504425856/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763946.174681:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763946.174690:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763946.174706:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3872 00000100:00080000:0.0:1789763946.174707:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763946.174716:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763946.174723:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:2.0:1789763946.174730:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763946.174738:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504425856:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789763946.174742:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763946.174762:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504426368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763946.174779:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763946.174783:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504425984:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763946.174803:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763946.174809:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763946.174818:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3872 00000800:00000100:0.0:1789763946.174826:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763946.174830:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763946.174840:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763946.174849:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763946.174853:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763946/real 1789763946] req@ffff9018d0f26d80 x1876699504425984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763946.174871:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504426496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763946.174879:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504425984/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763946.174929:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763946.174939:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763946.174945:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504425984:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763946.174972:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504426112:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763946.174998:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763946.175012:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763946/real 1789763946] req@ffff9018d0f25f80 x1876699504426112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763946.175026:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504426112/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763946.175096:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763946.175101:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763946.175105:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504426112:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763946.175113:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504426240:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763946.175163:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763946.175179:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763946/real 1789763946] req@ffff9018d0f27100 x1876699504426240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763946.175194:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504426240/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763946.175205:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763946.175209:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763946.175213:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504426240:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763946.175233:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504426368:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763946.175254:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763946.175266:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763946/real 1789763946] req@ffff9018d0f25180 x1876699504426368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763946.175280:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504426368/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763946.175290:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763946.175293:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763946.175296:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504426368:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763946.175304:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504426496:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763946.175322:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763946.175332:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763946/real 1789763946] req@ffff9018d0f24380 x1876699504426496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763946.175346:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504426496/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763946.175355:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763946.175359:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763946.175363:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504426496:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763946.814268:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763947.838307:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763948.862285:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763949.886291:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763950.910203:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763951.294141:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763951.294149:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763951.294152:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763951.294161:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3877 00000100:00080000:2.0:1789763951.294169:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763951.294175:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763951.294209:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504426624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763951.294237:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763951.294240:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763951.294242:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763951.294245:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3877 00000100:00080000:2.0:1789763951.294247:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763951.294250:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763951.294258:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763951.294266:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504426752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763951.294273:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763951.294276:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763951.294277:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763951.294280:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3877 00000100:00080000:2.0:1789763951.294282:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763951.294285:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763951.294290:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763951.294297:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504426880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763951.294302:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763951.294304:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763951.294305:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763951.294308:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3877 00000100:00080000:2.0:1789763951.294311:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763951.294314:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763951.294319:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763951.294328:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504426624:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763951.294329:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504427008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763951.294349:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763951.294352:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763951.294353:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763951.294357:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3877 00000100:00080000:2.0:1789763951.294360:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763951.294362:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763951.294367:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763951.294375:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504427136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763951.294382:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763951.294386:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763951.294387:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763951.294390:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3877 00000100:00080000:2.0:1789763951.294393:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763951.294395:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763951.294398:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763951.294404:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504427264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763951.294425:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763951.294473:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763951/real 1789763951] req@ffff9018d0f26300 x1876699504426624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763951.294488:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504426624/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763951.294498:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763951.294501:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763951.294505:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504426624:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763951.294511:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504426752:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763951.294526:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763951.294534:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763951/real 1789763951] req@ffff9018d0f24000 x1876699504426752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763951.294543:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504426752/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763951.294548:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763951.294550:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763951.294552:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504426752:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763951.294556:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504426880:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763951.294566:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763951.294572:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763951/real 1789763951] req@ffff9018d0f26680 x1876699504426880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763951.294580:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504426880/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763951.294585:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763951.294587:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763951.294588:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504426880:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763951.294615:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504427008:192.168.203.158@tcp:8:kworker.0 00000800:00000100:0.0:1789763951.294627:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763951.294634:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763951/real 1789763951] req@ffff9018d0f26a00 x1876699504427008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763951.294641:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504427008/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763951.294647:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763951.294648:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763951.294650:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504427008:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763951.294654:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504427136:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763951.294663:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763951.294670:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763951/real 1789763951] req@ffff9018d0f25c00 x1876699504427136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763951.294686:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504427136/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763951.294692:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763951.294694:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763951.294696:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504427136:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763951.294700:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504427264:192.168.203.158@tcp:38:kworker.0 00000800:00000100:0.0:1789763951.294711:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763951.294717:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763951/real 1789763951] req@ffff9018d0f27800 x1876699504427264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763951.294724:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504427264/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763951.294729:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763951.294731:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:0.0:1789763951.294733:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504427264:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763951.934252:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763952.958335:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763953.982212:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763955.006218:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763956.030325:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763956.414222:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763956.414234:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763956.414239:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763956.414251:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3882 00000100:00080000:0.0:1789763956.414266:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763956.414274:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763956.414323:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504427392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763956.414370:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763956.414379:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763956.414383:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763956.414392:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3882 00000100:00080000:0.0:1789763956.414400:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763956.414409:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763956.414428:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504427392:192.168.203.158@tcp:250:kworker.0 00000008:00100000:0.0:1789763956.414445:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763956.414482:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504427520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763956.414502:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763956.414506:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763956.414509:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763956.414514:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3882 00000100:00080000:0.0:1789763956.414521:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763956.414527:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763956.414535:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763956.414548:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504427648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763956.414560:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763956.414565:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763956.414568:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763956.414573:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3882 00000100:00080000:0.0:1789763956.414579:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763956.414583:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763956.414591:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763956.414607:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504427520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789763956.414614:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504427776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763956.414646:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763956.414654:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763956.414658:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763956.414665:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504427648:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763956.414686:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3882 00000100:00080000:0.0:1789763956.414718:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763956.414730:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763956.414742:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763956.414745:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504427776:192.168.203.158@tcp:8:kworker.0 00000100:00100000:0.0:1789763956.414767:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504427904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763956.414786:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763956.414791:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763956.414794:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763956.414804:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3882 00000100:00080000:0.0:1789763956.414812:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763956.414816:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504427904:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763956.414824:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763956.414840:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763956.414863:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504428032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763956.414908:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504428032:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763957.054315:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763958.078235:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763959.102251:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763959.486447:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:0.0:1789763959.486494:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763959.486527:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763959.486588:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763959.486605:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763959.486622:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:0.0:1789763959.486644:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789763959.486645:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763956/real 1789763959] req@ffff9018c8b9a680 x1876699504427776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1789763959.486660:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:1.0:1789763959.486692:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504427776/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763959.486711:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763959.486718:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763959.486723:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504427776:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763959.486733:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763956/real 1789763959] req@ffff9018c8b98700 x1876699504427520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763959.486755:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504427520/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763959.486767:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763959.486772:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763959.486777:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504427520:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763959.486786:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763956/real 1789763959] req@ffff9018c8b98a80 x1876699504427648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763959.486802:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504427648/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763959.486811:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763959.486816:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763959.486819:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504427648:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763959.486827:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763956/real 1789763959] req@ffff9018c8b9bb80 x1876699504427392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763959.486841:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504427392/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763959.486850:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763959.486854:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763959.486858:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504427392:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763959.486888:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763956/real 1789763959] req@ffff9018c8b9aa00 x1876699504428032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763959.486905:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504428032/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763959.486917:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763959.486922:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763959.486926:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504428032:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763959.486935:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763956/real 1789763959] req@ffff9018c8b98e00 x1876699504427904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763959.486951:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504427904/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763959.486962:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763959.486966:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763959.486971:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504427904:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763960.126228:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763961.150245:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763961.534226:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763961.534259:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763961.534265:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763961.534281:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3888 00000100:00080000:0.0:1789763961.534296:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763961.534325:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763961.534372:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504428160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763961.534413:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763961.534419:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763961.534421:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763961.534427:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3888 00000100:00080000:0.0:1789763961.534431:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763961.534437:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763961.534467:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763961.534484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504428288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763961.534499:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763961.534523:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789763961.534523:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504428160:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763961.534525:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763961.534535:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3888 00000100:00080000:0.0:1789763961.534540:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763961.534543:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763961.534548:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763961.534561:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504428416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763961.534571:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763961.534576:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763961.534578:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763961.534582:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3888 00000100:00080000:0.0:1789763961.534607:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763961.534613:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763961.534620:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789763961.534629:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789763961.534635:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504428544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763961.534647:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763961.534656:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763961.534658:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763961.534662:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3888 00000100:00080000:0.0:1789763961.534667:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763961.534672:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763961.534679:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763961.534691:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763961/real 1789763961] req@ffff9018c8b9b800 x1876699504428160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763961.534694:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504428672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763961.534723:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763961.534728:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763961.534731:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763961.534734:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504428160/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763961.534739:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3888 00000100:00080000:0.0:1789763961.534746:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763961.534753:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789763961.534756:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789763961.534759:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763961.534765:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763961.534770:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504428160:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763961.534778:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504428800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763961.534806:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504428288:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763961.534836:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763961.534850:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763961/real 1789763961] req@ffff9018c8b98000 x1876699504428288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763961.534862:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504428288/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763961.534871:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763961.534874:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763961.534877:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504428288:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763961.534883:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504428416:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763961.534902:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763961.534911:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763961/real 1789763961] req@ffff9018c8b99880 x1876699504428416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763961.534920:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504428416/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763961.534927:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763961.534930:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763961.534933:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504428416:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763961.534938:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504428544:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763961.534959:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763961.534986:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763961/real 1789763961] req@ffff9018c8b98380 x1876699504428544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763961.535004:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504428544/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763961.535028:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763961.535065:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763961.535069:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504428544:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763961.535078:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504428672:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763961.535102:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763961.535113:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763961/real 1789763961] req@ffff9018c8b9a300 x1876699504428672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763961.535125:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504428672/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763961.535135:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763961.535139:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763961.535143:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504428672:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763961.535152:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504428800:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763961.535175:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763961.535187:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763961/real 1789763961] req@ffff9018c8b9ad80 x1876699504428800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763961.535198:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504428800/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763961.535212:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763961.535215:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763961.535218:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504428800:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763962.174277:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763963.198244:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763964.222266:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763965.246233:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763966.270243:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763966.654231:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763966.654243:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763966.654249:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763966.654264:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3893 00000100:00080000:0.0:1789763966.654280:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763966.654292:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763966.654354:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504428928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763966.654390:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763966.654395:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763966.654397:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763966.654403:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3893 00000100:00080000:0.0:1789763966.654410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763966.654416:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763966.654430:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763966.654465:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504428928:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763966.654484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504429056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763966.654510:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763966.654518:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763966.654523:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763966.654538:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3893 00000100:00080000:0.0:1789763966.654550:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763966.654563:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789763966.654584:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789763966.654596:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763966.654652:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504429184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763966.654669:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763966/real 1789763966] req@ffff9018c8b99c00 x1876699504428928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763966.654671:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763966.654687:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763966.654693:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763966.654709:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3893 00000100:00080000:0.0:1789763966.654717:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763966.654725:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504428928/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763966.654730:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763966.654744:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1789763966.654752:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789763966.654770:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504429312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763966.654776:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763966.654794:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504428928:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763966.654836:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763966.654844:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763966.654849:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763966.654850:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504429056:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763966.654868:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3893 00000100:00080000:0.0:1789763966.654889:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763966.654901:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789763966.654911:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:0.0:1789763966.654914:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763966.654936:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763966/real 1789763966] req@ffff9018c8b99500 x1876699504429056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763966.654952:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504429440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763966.654967:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504429056/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763966.655021:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763966.655024:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763966.655089:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789763966.655090:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763966.655099:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763966.655104:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504429056:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763966.655114:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3893 00000100:00080000:0.0:1789763966.655123:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763966.655125:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504429184:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789763966.655136:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763966.655147:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763966.655172:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504429568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763966.655184:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763966.655230:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763966/real 1789763966] req@ffff9018c8b9ad80 x1876699504429184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763966.655267:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504429184/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763966.655291:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763966.655301:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763966.655309:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504429184:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763966.655329:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504429312:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763966.655374:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763966.655402:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763966/real 1789763966] req@ffff9018c8b9a300 x1876699504429312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763966.655430:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504429312/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763966.655457:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763966.655465:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763966.655477:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504429312:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763966.655529:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504429440:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763966.655551:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763966.655565:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763966/real 1789763966] req@ffff9018c8b98380 x1876699504429440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763966.655579:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504429440/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763966.655589:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763966.655592:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763966.655595:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504429440:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763966.655602:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504429568:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763966.655620:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763966.655631:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763966/real 1789763966] req@ffff9018c8b99880 x1876699504429568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763966.655644:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504429568/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763966.655662:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763966.655666:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763966.655670:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504429568:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763967.294312:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763968.318308:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763969.342172:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763970.366203:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763971.390152:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763971.774145:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763971.774153:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763971.774157:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763971.774165:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3898 00000100:00080000:0.0:1789763971.774175:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763971.774182:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789763971.774216:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504429696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763971.774235:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763971.774238:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763971.774239:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763971.774242:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3898 00000100:00080000:0.0:1789763971.774244:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763971.774247:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763971.774256:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763971.774269:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504429824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763971.774269:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504429696:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789763971.774283:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763971.774285:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763971.774287:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763971.774289:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3898 00000100:00080000:0.0:1789763971.774292:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763971.774295:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763971.774305:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763971.774315:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504429952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763971.774322:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763971.774324:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763971.774325:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763971.774328:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3898 00000100:00080000:0.0:1789763971.774330:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763971.774334:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:0.0:1789763971.774337:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763971.774343:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789763971.774351:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504430080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763971.774358:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763971.774360:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763971.774362:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763971.774365:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3898 00000100:00080000:0.0:1789763971.774368:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763971.774372:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763971/real 1789763971] req@ffff9018c8b98000 x1876699504429696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763971.774373:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763971.774379:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763971.774391:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504430208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763971.774397:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504429696/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763971.774402:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763971.774410:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789763971.774412:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763971.774417:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3898 00000100:00080000:2.0:1789763971.774418:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763971.774420:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763971.774424:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:0.0:1789763971.774426:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789763971.774431:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763971.774431:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504429696:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789763971.774445:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504430336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763971.774459:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504429824:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763971.774476:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763971.774485:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763971/real 1789763971] req@ffff9018c8b9b800 x1876699504429824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763971.774496:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504429824/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763971.774504:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763971.774508:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763971.774513:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504429824:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763971.774519:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504429952:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763971.774551:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763971.774565:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763971/real 1789763971] req@ffff9018c8b98e00 x1876699504429952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763971.774578:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504429952/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763971.774588:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763971.774592:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763971.774596:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504429952:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763971.774602:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504430080:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763971.774619:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763971.774627:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763971/real 1789763971] req@ffff9018c8b9aa00 x1876699504430080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763971.774635:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504430080/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763971.774641:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763971.774643:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763971.774646:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504430080:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763971.774650:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504430208:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763971.774660:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763971.774666:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763971/real 1789763971] req@ffff9018c8b9bb80 x1876699504430208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763971.774674:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504430208/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763971.774679:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763971.774681:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763971.774683:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504430208:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763971.774686:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504430336:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763971.774696:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763971.774701:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763971/real 1789763971] req@ffff9018c8b98a80 x1876699504430336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763971.774709:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504430336/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763971.774714:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763971.774715:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763971.774717:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504430336:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763972.414213:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763973.438213:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763974.462245:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763975.486142:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763976.510164:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763976.894114:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763976.894122:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763976.894125:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763976.894146:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3903 00000100:00080000:1.0:1789763976.894155:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763976.894160:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763976.894193:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5c00 x1876699504430464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763976.894222:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763976.894226:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763976.894228:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763976.894233:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3903 00000100:00080000:1.0:1789763976.894238:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763976.894243:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763976.894253:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763976.894262:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504430464:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789763976.894271:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6d80 x1876699504430592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763976.894283:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763976.894286:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763976.894295:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763976.894298:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3903 00000100:00080000:1.0:1789763976.894303:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763976.894306:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763976.894310:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763976.894320:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5f80 x1876699504430720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763976.894327:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763976.894330:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763976.894332:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789763976.894336:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763976.894340:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3903 00000100:00080000:1.0:1789763976.894344:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763976.894346:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763976.894350:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763976.894358:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4000 x1876699504430848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763976.894366:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763976.894369:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763976.894371:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763976.894374:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3903 00000100:00080000:1.0:1789763976.894377:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763976.894382:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763976.894384:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763976/real 1789763976] req@ffff9018c66a5c00 x1876699504430464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789763976.894385:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763976.894397:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7b80 x1876699504430976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763976.894404:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763976.894408:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763976.894411:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763976.894416:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3903 00000100:00080000:1.0:1789763976.894421:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763976.894426:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763976.894433:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763976.894441:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7800 x1876699504431104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763976.894445:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5c00 x1876699504430464/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763976.894463:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763976.894467:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763976.894471:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504430464:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763976.894494:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504430592:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763976.894519:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763976.894534:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763976/real 1789763976] req@ffff9018c66a6d80 x1876699504430592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763976.894549:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6d80 x1876699504430592/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763976.894556:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763976.894559:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763976.894561:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504430592:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763976.894566:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504430720:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763976.894582:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763976.894589:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763976/real 1789763976] req@ffff9018c66a5f80 x1876699504430720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763976.894597:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5f80 x1876699504430720/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763976.894607:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763976.894609:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763976.894611:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504430720:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763976.894615:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504430848:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763976.894634:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763976.894640:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763976/real 1789763976] req@ffff9018c66a4000 x1876699504430848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763976.894648:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4000 x1876699504430848/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763976.894653:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763976.894655:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763976.894657:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504430848:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763976.894661:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504430976:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763976.894671:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763976.894676:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763976/real 1789763976] req@ffff9018c66a7b80 x1876699504430976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763976.894684:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7b80 x1876699504430976/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763976.894688:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763976.894690:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763976.894692:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504430976:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763976.894696:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504431104:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763976.894705:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763976.894711:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763976/real 1789763976] req@ffff9018c66a7800 x1876699504431104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763976.894717:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7800 x1876699504431104/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763976.894730:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763976.894732:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763976.894734:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504431104:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763977.534172:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763978.558161:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763979.582191:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763980.606152:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763981.630154:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763982.014107:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763982.014115:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763982.014120:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763982.014128:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3908 00000100:00080000:1.0:1789763982.014137:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763982.014143:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789763982.014175:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4e00 x1876699504431232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763982.014198:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763982.014202:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763982.014203:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763982.014206:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3908 00000100:00080000:1.0:1789763982.014209:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763982.014212:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763982.014220:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763982.014232:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4700 x1876699504431360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763982.014243:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504431232:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789763982.014246:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763982.014252:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763982.014254:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763982.014259:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3908 00000100:00080000:1.0:1789763982.014263:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763982.014269:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763982.014275:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763982.014289:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6680 x1876699504431488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763982.014297:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763982.014300:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763982.014301:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763982.014308:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3908 00000800:00000100:2.0:1789763982.014309:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789763982.014314:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763982.014319:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763982.014323:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789763982.014332:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5500 x1876699504431616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763982.014348:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763982.014353:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789763982.014354:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763982/real 1789763982] req@ffff9018c66a4e00 x1876699504431232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763982.014355:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763982.014362:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3908 00000100:00080000:1.0:1789763982.014364:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763982.014366:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789763982.014372:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763982.014386:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5180 x1876699504431744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763982.014392:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4e00 x1876699504431232/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763982.014397:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763982.014407:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789763982.014410:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763982.014412:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763982.014415:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3908 00000100:00080000:1.0:1789763982.014419:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763982.014419:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789763982.014427:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763982.014428:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504431232:192.168.203.158@tcp:250:kworker.0 00000008:00100000:1.0:1789763982.014431:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763982.014441:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6300 x1876699504431872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763982.014455:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504431360:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763982.014483:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763982.014497:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763982/real 1789763982] req@ffff9018c66a4700 x1876699504431360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763982.014511:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4700 x1876699504431360/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763982.014520:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763982.014523:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763982.014525:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504431360:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763982.014530:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504431488:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763982.014545:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763982.014552:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763982/real 1789763982] req@ffff9018c66a6680 x1876699504431488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763982.014560:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6680 x1876699504431488/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763982.014565:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763982.014567:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763982.014569:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504431488:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763982.014572:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504431616:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763982.014582:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763982.014588:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763982/real 1789763982] req@ffff9018c66a5500 x1876699504431616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763982.014599:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5500 x1876699504431616/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763982.014605:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763982.014606:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763982.014608:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504431616:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763982.014611:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504431744:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763982.014621:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763982.014626:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763982/real 1789763982] req@ffff9018c66a5180 x1876699504431744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763982.014633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5180 x1876699504431744/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763982.014638:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763982.014640:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763982.014641:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504431744:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763982.014645:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504431872:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763982.014654:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763982.014659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763982/real 1789763982] req@ffff9018c66a6300 x1876699504431872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763982.014666:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6300 x1876699504431872/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763982.014675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763982.014677:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763982.014679:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504431872:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763982.654174:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763983.678137:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763984.702194:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763985.726225:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763986.750150:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763987.134117:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763987.134126:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763987.134129:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763987.134137:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3913 00000100:00080000:3.0:1789763987.134146:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763987.134151:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763987.134182:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e680 x1876699504432000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763987.134202:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763987.134205:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763987.134206:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763987.134209:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3913 00000100:00080000:3.0:1789763987.134212:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763987.134214:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763987.134222:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763987.134231:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070df80 x1876699504432128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763987.134239:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504432000:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763987.134240:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763987.134248:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763987.134249:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763987.134252:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3913 00000100:00080000:3.0:1789763987.134254:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763987.134256:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763987.134260:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763987.134267:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070ca80 x1876699504432256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763987.134273:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763987.134275:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763987.134276:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763987.134279:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3913 00000100:00080000:3.0:1789763987.134291:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763987.134293:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763987.134297:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789763987.134301:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:3.0:1789763987.134309:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504432384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763987.134318:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763987.134320:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763987.134321:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763987.134324:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3913 00000100:00080000:3.0:1789763987.134326:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763987.134330:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763987.134334:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763987.134335:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763987/real 1789763987] req@ffff9018d070e680 x1876699504432000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763987.134344:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504432512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763987.134370:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763987.134374:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789763987.134375:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763987.134378:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3913 00000100:00100000:2.0:1789763987.134382:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e680 x1876699504432000/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763987.134382:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763987.134387:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763987.134394:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763987.134399:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763987.134405:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763987.134413:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504432000:192.168.203.158@tcp:250:kworker.0 00000100:00100000:3.0:1789763987.134414:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504432640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763987.134439:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504432128:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763987.134456:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763987.134465:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763987/real 1789763987] req@ffff9018d070df80 x1876699504432128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763987.134474:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070df80 x1876699504432128/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763987.134483:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763987.134486:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763987.134488:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504432128:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763987.134491:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504432256:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763987.134502:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763987.134507:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763987/real 1789763987] req@ffff9018d070ca80 x1876699504432256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763987.134514:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070ca80 x1876699504432256/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763987.134519:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763987.134521:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763987.134523:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504432256:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763987.134527:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504432384:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763987.134536:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763987.134542:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763987/real 1789763987] req@ffff9018d070d500 x1876699504432384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763987.134548:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504432384/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763987.134553:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763987.134555:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763987.134557:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504432384:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763987.134561:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504432512:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763987.134570:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763987.134576:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763987/real 1789763987] req@ffff9018d070dc00 x1876699504432512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763987.134583:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504432512/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763987.134588:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763987.134589:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763987.134591:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504432512:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763987.134594:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504432640:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763987.134603:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763987.134609:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763987/real 1789763987] req@ffff9018d070c000 x1876699504432640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763987.134620:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504432640/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763987.134627:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763987.134630:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763987.134634:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504432640:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763987.774176:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763988.798176:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763989.822195:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763990.846195:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763991.870252:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763992.254121:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763992.254132:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763992.254136:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763992.254147:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3918 00000100:00080000:3.0:1789763992.254157:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763992.254164:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:3.0:1789763992.254197:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070e300 x1876699504432768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763992.254220:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763992.254226:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763992.254228:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763992.254231:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3918 00000100:00080000:3.0:1789763992.254234:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763992.254237:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763992.254244:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763992.254253:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f100 x1876699504432896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763992.254262:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763992.254266:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789763992.254270:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504432768:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763992.254270:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763992.254280:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3918 00000100:00080000:3.0:1789763992.254285:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763992.254289:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763992.254294:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763992.254309:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070f800 x1876699504433024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763992.254318:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763992.254321:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763992.254323:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763992.254326:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3918 00000100:00080000:3.0:1789763992.254329:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763992.254331:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000800:00000100:2.0:1789763992.254337:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000008:00100000:3.0:1789763992.254337:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789763992.254364:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070c000 x1876699504433152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763992.254372:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763992.254375:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763992.254376:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763992.254380:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3918 00000100:00100000:2.0:1789763992.254382:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763992/real 1789763992] req@ffff9018d070e300 x1876699504432768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763992.254383:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763992.254390:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:3.0:1789763992.254397:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763992.254407:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070e300 x1876699504432768/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763992.254408:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070dc00 x1876699504433280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763992.254437:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763992.254439:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763992.254442:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:3.0:1789763992.254446:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789763992.254449:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504432768:192.168.203.158@tcp:250:kworker.0 00000100:00080000:3.0:1789763992.254451:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763992.254458:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3918 00000100:00080000:3.0:1789763992.254463:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763992.254467:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763992.254472:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504432896:192.168.203.158@tcp:38:kworker.0 00000008:00100000:3.0:1789763992.254475:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763992.254491:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d070d500 x1876699504433408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763992.254498:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763992.254512:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763992/real 1789763992] req@ffff9018d070f100 x1876699504432896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763992.254523:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f100 x1876699504432896/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763992.254529:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763992.254532:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763992.254534:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504432896:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763992.254537:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504433024:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763992.254548:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763992.254555:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763992/real 1789763992] req@ffff9018d070f800 x1876699504433024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763992.254562:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070f800 x1876699504433024/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763992.254567:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763992.254569:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763992.254578:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504433024:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763992.254583:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504433152:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789763992.254595:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763992.254601:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763992/real 1789763992] req@ffff9018d070c000 x1876699504433152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763992.254610:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070c000 x1876699504433152/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763992.254618:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763992.254621:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763992.254624:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504433152:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789763992.254630:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504433280:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763992.254648:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763992.254659:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763992/real 1789763992] req@ffff9018d070dc00 x1876699504433280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763992.254669:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070dc00 x1876699504433280/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763992.254678:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763992.254681:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763992.254684:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504433280:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789763992.254702:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504433408:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789763992.254721:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763992.254732:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763992/real 1789763992] req@ffff9018d070d500 x1876699504433408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763992.254742:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d070d500 x1876699504433408/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763992.254750:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763992.254753:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763992.254755:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d070d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504433408:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789763992.894176:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763993.918251:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763994.942264:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763995.966251:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763996.990235:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763997.374190:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763997.374199:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763997.374202:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763997.374210:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3923 00000100:00080000:2.0:1789763997.374220:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763997.374227:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789763997.374262:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504433536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763997.374287:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763997.374291:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763997.374293:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763997.374298:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3923 00000100:00080000:2.0:1789763997.374302:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763997.374306:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763997.374317:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763997.374379:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504433536:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789763997.374392:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24a80 x1876699504433664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763997.374403:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763997.374406:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763997.374408:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763997.374412:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3923 00000100:00080000:2.0:1789763997.374415:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763997.374419:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763997.374425:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763997.374439:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25500 x1876699504433792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763997.374452:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763997.374454:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763997.374457:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763997.374460:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3923 00000800:00000100:1.0:1789763997.374466:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789763997.374469:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763997.374474:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763997.374482:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789763997.374503:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27480 x1876699504433920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763997.374517:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763997.374518:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763997/real 1789763997] req@ffff9018d0f24700 x1876699504433536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763997.374539:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763997.374542:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763997.374550:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3923 00000100:00100000:1.0:1789763997.374552:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504433536/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763997.374555:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763997.374567:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763997.374579:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763997.374583:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763997.374588:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789763997.374592:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24e00 x1876699504434048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763997.374594:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504433536:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789763997.374621:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763997.374626:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789763997.374628:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763997.374634:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3923 00000100:00100000:1.0:1789763997.374638:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504433664:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789763997.374639:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763997.374651:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789763997.374660:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763997.374667:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789763997.374680:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504434176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763997.374692:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763997/real 1789763997] req@ffff9018d0f24a80 x1876699504433664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763997.374764:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24a80 x1876699504433664/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763997.374775:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763997.374778:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763997.374782:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504433664:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763997.374788:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504433792:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763997.374810:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763997.374820:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763997/real 1789763997] req@ffff9018d0f25500 x1876699504433792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763997.374828:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25500 x1876699504433792/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763997.374833:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763997.374835:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763997.374837:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504433792:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763997.374842:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504433920:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789763997.374854:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763997.374860:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763997/real 1789763997] req@ffff9018d0f27480 x1876699504433920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763997.374867:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27480 x1876699504433920/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763997.374873:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763997.374874:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763997.374876:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504433920:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789763997.374880:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504434048:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763997.374891:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763997.374897:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763997/real 1789763997] req@ffff9018d0f24e00 x1876699504434048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763997.374904:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24e00 x1876699504434048/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763997.374909:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763997.374910:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763997.374912:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504434048:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789763997.374926:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504434176:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789763997.374937:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789763997.374943:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763997/real 1789763997] req@ffff9018d0f24700 x1876699504434176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763997.374952:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504434176/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763997.374960:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763997.374963:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789763997.374967:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504434176:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789763998.014193:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763999.038190:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764000.062177:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764001.086174:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764002.110145:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789764002.494146:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764002.494155:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789764002.494159:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764002.494169:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3929 00000100:00080000:2.0:1789764002.494180:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764002.494189:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789764002.494223:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504434304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764002.494246:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764002.494249:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789764002.494251:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764002.494255:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3929 00000100:00080000:2.0:1789764002.494260:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764002.494264:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764002.494272:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789764002.494278:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504434304:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789764002.494285:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504434432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764002.494295:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764002.494298:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789764002.494300:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764002.494304:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3929 00000100:00080000:2.0:1789764002.494307:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764002.494310:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764002.494314:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764002.494323:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504434560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764002.494329:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764002.494332:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789764002.494333:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789764002.494338:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789764002.494342:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3929 00000100:00080000:2.0:1789764002.494346:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764002.494350:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764002.494354:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789764002.494362:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504434688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789764002.494375:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764002/real 1789764002] req@ffff9018d0f27800 x1876699504434304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764002.494381:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764002.494387:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789764002.494388:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789764002.494393:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504434304/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764002.494393:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3929 00000100:00080000:2.0:1789764002.494396:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764002.494403:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789764002.494410:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789764002.494410:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789764002.494418:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764002.494424:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504434304:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789764002.494426:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504434816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764002.494435:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764002.494438:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789764002.494439:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764002.494442:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3929 00000100:00080000:2.0:1789764002.494445:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789764002.494449:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504434432:192.168.203.158@tcp:38:kworker.0 00000100:00080000:2.0:1789764002.494451:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764002.494457:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764002.494467:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504434944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789764002.494487:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764002.494505:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764002/real 1789764002] req@ffff9018d0f25c00 x1876699504434432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764002.494515:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504434432/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764002.494527:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764002.494530:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764002.494532:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504434432:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764002.494537:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504434560:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764002.494552:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764002.494560:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764002/real 1789764002] req@ffff9018d0f26a00 x1876699504434560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764002.494568:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504434560/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764002.494574:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764002.494577:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764002.494579:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504434560:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764002.494583:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504434688:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789764002.494597:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764002.494607:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764002/real 1789764002] req@ffff9018d0f26680 x1876699504434688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764002.494617:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504434688/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764002.494623:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764002.494625:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764002.494628:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504434688:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789764002.494631:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504434816:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764002.494642:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764002.494654:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764002/real 1789764002] req@ffff9018d0f24000 x1876699504434816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764002.494664:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504434816/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764002.494669:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764002.494671:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764002.494675:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504434816:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764002.494708:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504434944:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764002.494740:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764002.494749:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764002/real 1789764002] req@ffff9018d0f26300 x1876699504434944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764002.494757:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504434944/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764002.494762:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764002.494764:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764002.494766:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504434944:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789764003.134136:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764004.158159:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764005.182171:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764006.206192:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764007.230243:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789764007.614171:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764007.614181:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789764007.614185:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764007.614195:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3934 00000100:00080000:0.0:1789764007.614205:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764007.614211:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789764007.614247:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98700 x1876699504435072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764007.614273:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764007.614276:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789764007.614277:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764007.614281:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3934 00000100:00080000:0.0:1789764007.614283:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764007.614285:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764007.614293:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789764007.614310:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a680 x1876699504435200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764007.614318:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764007.614320:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789764007.614322:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764007.614326:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3934 00000100:00080000:0.0:1789764007.614330:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789764007.614330:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504435072:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789764007.614335:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764007.614342:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789764007.614349:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b100 x1876699504435328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764007.614355:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764007.614357:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789764007.614359:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764007.614361:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3934 00000100:00080000:0.0:1789764007.614364:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764007.614366:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764007.614369:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789764007.614378:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99f80 x1876699504435456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764007.614384:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764007.614387:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789764007.614388:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764007.614391:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3934 00000100:00080000:0.0:1789764007.614394:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764007.614397:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764007.614399:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789764007.614407:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789764007.614409:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b480 x1876699504435584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764007.614422:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764007.614425:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789764007.614427:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764007.614429:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3934 00000100:00080000:0.0:1789764007.614432:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764007.614434:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764007.614437:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789764007.614445:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98a80 x1876699504435712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789764007.614453:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764007/real 1789764007] req@ffff9018c8b98700 x1876699504435072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764007.614468:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98700 x1876699504435072/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764007.614476:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764007.614480:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764007.614483:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504435072:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789764007.614502:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504435200:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764007.614518:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764007.614527:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764007/real 1789764007] req@ffff9018c8b9a680 x1876699504435200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764007.614536:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a680 x1876699504435200/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764007.614541:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764007.614544:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764007.614546:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504435200:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764007.614549:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504435328:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764007.614559:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764007.614564:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764007/real 1789764007] req@ffff9018c8b9b100 x1876699504435328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764007.614572:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b100 x1876699504435328/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764007.614582:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764007.614583:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764007.614585:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504435328:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764007.614589:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504435456:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789764007.614599:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764007.614605:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764007/real 1789764007] req@ffff9018c8b99f80 x1876699504435456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764007.614612:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99f80 x1876699504435456/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764007.614616:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764007.614618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764007.614620:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504435456:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789764007.614623:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504435584:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764007.614633:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764007.614638:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764007/real 1789764007] req@ffff9018c8b9b480 x1876699504435584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764007.614645:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b480 x1876699504435584/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764007.614650:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764007.614652:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764007.614654:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504435584:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764007.614658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504435712:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764007.614669:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764007.614688:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764007/real 1789764007] req@ffff9018c8b98a80 x1876699504435712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764007.614697:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98a80 x1876699504435712/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764007.614702:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764007.614704:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764007.614706:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504435712:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789764008.254228:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764009.278272:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764010.302194:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764011.326317:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764012.350271:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789764012.734210:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764012.734222:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789764012.734226:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764012.734239:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3939 00000100:00080000:0.0:1789764012.734252:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764012.734260:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789764012.734311:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9bb80 x1876699504435840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764012.734333:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764012.734336:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789764012.734338:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764012.734341:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3939 00000100:00080000:0.0:1789764012.734343:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764012.734346:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764012.734357:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789764012.734368:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9aa00 x1876699504435968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764012.734377:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764012.734379:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789764012.734381:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764012.734386:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3939 00000100:00100000:1.0:1789764012.734386:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504435840:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789764012.734391:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764012.734397:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764012.734402:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789764012.734417:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98e00 x1876699504436096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764012.734429:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764012.734433:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789764012.734435:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764012.734440:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3939 00000100:00080000:0.0:1789764012.734445:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764012.734450:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764012.734456:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789764012.734461:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:0.0:1789764012.734469:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9b800 x1876699504436224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764012.734479:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764012.734481:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789764012.734483:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764012.734486:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3939 00000100:00080000:0.0:1789764012.734489:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764012.734493:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764012.734497:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789764012.734507:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98000 x1876699504436352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789764012.734513:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764012/real 1789764012] req@ffff9018c8b9bb80 x1876699504435840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789764012.734518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764012.734526:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789764012.734529:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764012.734534:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3939 00000100:00080000:0.0:1789764012.734537:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789764012.734540:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9bb80 x1876699504435840/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789764012.734551:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00080000:1.0:1789764012.734552:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789764012.734558:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789764012.734559:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764012.734566:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504435840:192.168.203.158@tcp:250:kworker.0 00000100:00100000:0.0:1789764012.734571:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504436480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789764012.734590:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504435968:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764012.734611:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764012.734621:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764012/real 1789764012] req@ffff9018c8b9aa00 x1876699504435968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764012.734629:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9aa00 x1876699504435968/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764012.734634:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764012.734650:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764012.734653:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504435968:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764012.734658:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504436096:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764012.734674:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764012.734681:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764012/real 1789764012] req@ffff9018c8b98e00 x1876699504436096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764012.734689:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98e00 x1876699504436096/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764012.734694:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764012.734696:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764012.734698:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504436096:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764012.734702:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504436224:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789764012.734713:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764012.734719:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764012/real 1789764012] req@ffff9018c8b9b800 x1876699504436224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764012.734726:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9b800 x1876699504436224/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764012.734732:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764012.734734:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764012.734737:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504436224:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789764012.734740:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504436352:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764012.734750:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764012.734755:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764012/real 1789764012] req@ffff9018c8b98000 x1876699504436352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764012.734762:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98000 x1876699504436352/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764012.734767:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764012.734769:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764012.734771:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504436352:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764012.734774:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504436480:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764012.734784:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764012.734789:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764012/real 1789764012] req@ffff9018c8b99880 x1876699504436480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764012.734796:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504436480/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764012.734801:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764012.734802:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764012.734805:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504436480:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789764013.374279:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764014.398202:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764015.422238:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764016.446202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764017.470190:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789764017.854130:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789764017.854140:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789764017.854144:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789764017.854155:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3944 00000100:00080000:1.0:1789764017.854166:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789764017.854173:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789764017.854216:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504436608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789764017.854243:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789764017.854247:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789764017.854249:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789764017.854253:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3944 00000100:00080000:1.0:1789764017.854257:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789764017.854261:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789764017.854270:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789764017.854283:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504436736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789764017.854288:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504436608:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789764017.854304:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789764017.854307:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789764017.854310:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789764017.854315:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3944 00000100:00080000:1.0:1789764017.854319:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789764017.854323:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789764017.854329:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789764017.854341:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504436864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789764017.854351:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789764017.854354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789764017.854359:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789764017.854359:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789764017.854368:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3944 00000100:00080000:1.0:1789764017.854373:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789764017.854377:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789764017.854384:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789764017.854396:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504436992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789764017.854405:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764017/real 1789764017] req@ffff9018c66a5880 x1876699504436608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764017.854409:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789764017.854419:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789764017.854423:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789764017.854429:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3944 00000100:00080000:1.0:1789764017.854433:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789764017.854438:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504436608/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764017.854441:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789764017.854450:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789764017.854465:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504437120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764017.854466:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764017.854476:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00080000:1.0:1789764017.854482:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789764017.854486:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504436608:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789764017.854489:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789764017.854493:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789764017.854499:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3944 00000100:00080000:1.0:1789764017.854503:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789764017.854507:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789764017.854511:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764017.854517:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504436736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764017.854524:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504437248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789764017.854549:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789764017.854565:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764017/real 1789764017] req@ffff9018c66a7480 x1876699504436736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764017.854578:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504436736/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764017.854588:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764017.854591:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764017.854595:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504436736:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789764017.854614:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504436864:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789764017.854635:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789764017.854646:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764017/real 1789764017] req@ffff9018c66a4a80 x1876699504436864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764017.854657:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504436864/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764017.854666:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764017.854670:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764017.854673:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504436864:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789764017.854681:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504436992:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789764017.854701:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789764017.854712:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764017/real 1789764017] req@ffff9018c66a7100 x1876699504436992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764017.854723:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504436992/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764017.854732:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764017.854735:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764017.854739:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504436992:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789764017.854745:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504437120:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789764017.854764:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789764017.854776:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764017/real 1789764017] req@ffff9018c66a4380 x1876699504437120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764017.854787:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504437120/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764017.854796:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764017.854799:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764017.854802:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504437120:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789764017.854821:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504437248:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789764017.854842:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789764017.854854:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764017/real 1789764017] req@ffff9018c66a6a00 x1876699504437248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764017.854867:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504437248/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764017.854883:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764017.854886:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764017.854890:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504437248:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789764018.494616:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764019.518228:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764020.542165:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764021.566234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764022.590183:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789764022.974225:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764022.974234:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789764022.974238:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764022.974257:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3949 00000100:00080000:0.0:1789764022.974265:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764022.974271:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:0.0:1789764022.974307:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b98380 x1876699504437376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764022.974339:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764022.974342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789764022.974349:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764022.974352:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3949 00000100:00080000:0.0:1789764022.974354:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764022.974357:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764022.974366:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789764022.974380:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9a300 x1876699504437504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789764022.974388:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504437376:192.168.203.158@tcp:250:kworker.0 00000100:00080000:0.0:1789764022.974391:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764022.974398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789764022.974400:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764022.974403:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3949 00000100:00080000:0.0:1789764022.974409:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764022.974412:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764022.974416:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789764022.974426:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b9ad80 x1876699504437632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764022.974433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764022.974436:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789764022.974437:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764022.974439:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3949 00000100:00080000:0.0:1789764022.974442:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764022.974445:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764022.974448:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789764022.974456:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99500 x1876699504437760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789764022.974463:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764022.974465:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789764022.974466:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764022.974469:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3949 00000100:00080000:0.0:1789764022.974485:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764022.974490:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:0.0:1789764022.974496:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764022.974496:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504437504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:0.0:1789764022.974517:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99c00 x1876699504437888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789764022.974528:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504437632:192.168.203.158@tcp:38:kworker.0 00000100:00080000:0.0:1789764022.974530:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789764022.974536:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789764022.974549:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504437760:192.168.203.158@tcp:8:kworker.0 00000100:00080000:0.0:1789764022.974555:0:468:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789764022.974576:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3949 00000100:00080000:0.0:1789764022.974583:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789764022.974589:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789764022.974590:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504437888:192.168.203.158@tcp:38:kworker.0 00000008:00100000:0.0:1789764022.974598:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789764022.974617:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c8b99880 x1876699504438016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789764022.974702:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504438016:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789764023.614256:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764024.638202:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764025.662216:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789764026.046286:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.158:988 00000400:00000100:3.0:1789764026.046321:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.158@tcp at host 192.168.203.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789764026.046334:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789764026.046367:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789764026.046376:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789764026.046385:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789764026.046392:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000800:00000100:3.0:1789764026.046400:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.158@tcp 00000100:00100000:2.0:1789764026.046407:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764022/real 1789764026] req@ffff9018c8b9a300 x1876699504437504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764026.046438:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9a300 x1876699504437504/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764026.046450:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764026.046454:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764026.046459:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504437504:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789764026.046466:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764022/real 1789764026] req@ffff9018c8b9ad80 x1876699504437632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764026.046478:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b9ad80 x1876699504437632/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764026.046486:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764026.046489:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764026.046492:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b9ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504437632:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789764026.046498:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764022/real 1789764026] req@ffff9018c8b99500 x1876699504437760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764026.046508:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99500 x1876699504437760/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764026.046515:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764026.046517:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764026.046520:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504437760:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789764026.046526:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764022/real 1789764026] req@ffff9018c8b98380 x1876699504437376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764026.046536:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b98380 x1876699504437376/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764026.046566:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764026.046570:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764026.046573:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504437376:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789764026.046597:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764022/real 1789764026] req@ffff9018c8b99880 x1876699504438016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764026.046607:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99880 x1876699504438016/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764026.046613:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764026.046617:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764026.046620:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504438016:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789764026.046625:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764022/real 1789764026] req@ffff9018c8b99c00 x1876699504437888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764026.046633:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c8b99c00 x1876699504437888/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764026.046639:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764026.046642:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764026.046645:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c8b99c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504437888:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789764026.686200:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764027.710179:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789764028.094172:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789764028.094179:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789764028.094182:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789764028.094190:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3954 00000100:00080000:1.0:1789764028.094198:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789764028.094203:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789764028.094228:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a6a00 x1876699504438144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789764028.094245:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789764028.094248:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789764028.094249:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789764028.094251:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3954 00000100:00080000:1.0:1789764028.094253:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789764028.094256:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789764028.094262:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789764028.094270:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4380 x1876699504438272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789764028.094276:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504438144:192.168.203.158@tcp:250:kworker.0 00000100:00080000:1.0:1789764028.094280:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789764028.094285:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789764028.094286:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789764028.094289:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3954 00000100:00080000:1.0:1789764028.094291:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789764028.094292:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789764028.094295:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789764028.094301:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7100 x1876699504438400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789764028.094307:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789764028.094309:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789764028.094310:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789764028.094312:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3954 00000100:00080000:1.0:1789764028.094314:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789764028.094317:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789764028.094321:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789764028.094331:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a4a80 x1876699504438528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789764028.094340:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789764028.094344:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789764028.094347:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789764028.094347:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:1.0:1789764028.094355:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3954 00000100:00080000:1.0:1789764028.094360:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789764028.094363:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789764028.094368:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789764028.094379:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a7480 x1876699504438656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789764028.094386:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764028/real 1789764028] req@ffff9018c66a6a00 x1876699504438144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764028.094391:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789764028.094399:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789764028.094403:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789764028.094409:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3954 00000100:00100000:2.0:1789764028.094414:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a6a00 x1876699504438144/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764028.094415:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789764028.094423:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:1.0:1789764028.094432:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789764028.094433:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764028.094439:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764028.094446:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018c66a5880 x1876699504438784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789764028.094447:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504438144:192.168.203.158@tcp:250:kworker.0 00000100:00100000:2.0:1789764028.094474:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504438272:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789764028.094492:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789764028.094502:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764028/real 1789764028] req@ffff9018c66a4380 x1876699504438272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764028.094513:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4380 x1876699504438272/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764028.094520:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764028.095145:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764028.095152:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504438272:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789764028.095159:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504438400:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789764028.095181:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789764028.095192:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764028/real 1789764028] req@ffff9018c66a7100 x1876699504438400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764028.095205:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7100 x1876699504438400/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764028.095219:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764028.095222:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764028.095225:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504438400:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789764028.095231:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504438528:192.168.203.158@tcp:8:kworker.0 00000800:00000100:2.0:1789764028.095246:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789764028.095255:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764028/real 1789764028] req@ffff9018c66a4a80 x1876699504438528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764028.095266:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a4a80 x1876699504438528/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764028.095273:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764028.095276:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764028.095279:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504438528:192.168.203.158@tcp:8:kworker.0 00000100:00100000:2.0:1789764028.095285:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504438656:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789764028.095313:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789764028.095322:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764028/real 1789764028] req@ffff9018c66a7480 x1876699504438656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764028.095333:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a7480 x1876699504438656/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764028.095340:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764028.095343:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764028.095346:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504438656:192.168.203.158@tcp:38:kworker.0 00000100:00100000:2.0:1789764028.095352:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504438784:192.168.203.158@tcp:38:kworker.0 00000800:00000100:2.0:1789764028.095370:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:2.0:1789764028.095380:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764028/real 1789764028] req@ffff9018c66a5880 x1876699504438784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764028.095391:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018c66a5880 x1876699504438784/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764028.095398:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789764028.095401:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:2.0:1789764028.095404:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018c66a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504438784:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789764028.734202:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764029.758206:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764030.782234:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764031.806213:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764032.830169:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789764033.214140:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764033.214150:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789764033.214154:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764033.214164:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3959 00000100:00080000:2.0:1789764033.214174:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764033.214179:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789764033.214213:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24380 x1876699504438912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764033.214233:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764033.214235:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789764033.214237:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764033.214240:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3959 00000100:00080000:2.0:1789764033.214242:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764033.214244:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764033.214251:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764033.214259:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25180 x1876699504439040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764033.214266:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764033.214270:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789764033.214271:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504438912:192.168.203.158@tcp:250:kworker.0 00000100:00080000:2.0:1789764033.214271:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764033.214278:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3959 00000100:00080000:2.0:1789764033.214281:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764033.214283:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764033.214285:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764033.214293:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27100 x1876699504439168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764033.214299:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764033.214302:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789764033.214303:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764033.214305:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3959 00000100:00080000:2.0:1789764033.214307:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764033.214309:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764033.214312:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789764033.214319:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25f80 x1876699504439296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789764033.214324:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00080000:2.0:1789764033.214331:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764033.214334:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789764033.214335:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764033.214338:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3959 00000100:00080000:2.0:1789764033.214342:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764033.214344:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764033.214346:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764033.214354:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26d80 x1876699504439424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789764033.214359:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764033/real 1789764033] req@ffff9018d0f24380 x1876699504438912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764033.214362:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764033.214368:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789764033.214371:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764033.214377:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3959 00000100:00080000:2.0:1789764033.214380:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764033.214382:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764033.214385:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789764033.214386:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24380 x1876699504438912/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764033.214394:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26300 x1876699504439552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789764033.214396:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764033.214403:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764033.214409:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504438912:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789764033.214426:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504439040:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764033.214440:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764033.214447:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764033/real 1789764033] req@ffff9018d0f25180 x1876699504439040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764033.214454:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25180 x1876699504439040/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764033.214459:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764033.214460:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764033.214462:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504439040:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764033.214465:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504439168:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764033.214482:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764033.214506:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764033/real 1789764033] req@ffff9018d0f27100 x1876699504439168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764033.214513:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27100 x1876699504439168/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764033.214517:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764033.214519:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764033.214521:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504439168:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764033.214524:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504439296:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789764033.214536:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764033.214542:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764033/real 1789764033] req@ffff9018d0f25f80 x1876699504439296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764033.214552:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25f80 x1876699504439296/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764033.214559:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764033.214562:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764033.214565:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504439296:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789764033.214570:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504439424:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764033.214588:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764033.214598:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764033/real 1789764033] req@ffff9018d0f26d80 x1876699504439424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764033.214608:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26d80 x1876699504439424/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764033.214615:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764033.214618:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764033.214621:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504439424:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764033.214626:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504439552:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764033.214643:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764033.214651:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764033/real 1789764033] req@ffff9018d0f26300 x1876699504439552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764033.214662:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26300 x1876699504439552/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764033.214675:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764033.214677:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764033.214680:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504439552:192.168.203.158@tcp:38:kworker.0 00010000:00010000:2.0:1789764033.854187:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764034.878174:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764035.902205:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764036.926238:0:12526:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789764037.950253:0:6165:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789764038.334162:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8b68caea-910d-4e7e-8105-d8da387b4d8a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764038.334173:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789764038.334178:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764038.334188:0:12619:0:(import.c:510:import_select_connection()) MGC192.168.203.158@tcp: connect to NID 192.168.203.158@tcp last attempt 3964 00000100:00080000:2.0:1789764038.334199:0:12619:0:(import.c:589:import_select_connection()) MGC192.168.203.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764038.334207:0:12619:0:(import.c:624:import_select_connection()) MGC192.168.203.158@tcp: import ffff9018c8859000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:2.0:1789764038.334245:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24000 x1876699504439680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764038.334272:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764038.334276:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789764038.334279:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764038.334284:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3964 00000100:00080000:2.0:1789764038.334288:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764038.334293:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c54e9800: import ffff9018c855e800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764038.334304:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764038.334319:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26680 x1876699504439808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764038.334331:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764038.334334:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789764038.334336:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764038.334341:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3964 00000100:00080000:2.0:1789764038.334345:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764038.334351:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c54e9800: import ffff9018c855a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764038.334356:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764038.334370:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f26a00 x1876699504439936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764038.334378:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 79662a0f-af46-4cf0-8a56-ded8e9c54f3e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764038.334381:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789764038.334382:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764038.334386:0:12619:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: connect to NID 192.168.203.158@tcp last attempt 3964 00000100:00080000:2.0:1789764038.334390:0:12619:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764038.334393:0:12619:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9018c54e9800: import ffff9018c855c000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764038.334398:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789764038.334407:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f25c00 x1876699504440064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764038.334415:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764038.334418:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789764038.334419:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764038.334423:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3964 00000100:00080000:2.0:1789764038.334426:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764038.334430:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9018c5884000: import ffff9018c888a000 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000100:00100000:1.0:1789764038.334432:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504439680:192.168.203.158@tcp:250:kworker.0 00000008:00100000:2.0:1789764038.334434:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764038.334448:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f27800 x1876699504440192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764038.334488:0:12619:0:(pinger.c:229:ptlrpc_pinger_process_import()) 453b2994-2b6d-4a69-9aac-a9e10430c708->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764038.334491:0:12619:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789764038.334492:0:12619:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764038.334497:0:12619:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: connect to NID 192.168.203.158@tcp last attempt 3964 00000100:00080000:2.0:1789764038.334500:0:12619:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764038.334504:0:12619:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9018c5884000: import ffff9018c7fd9800 using connection 192.168.203.158@tcp/192.168.203.158@tcp 00000008:00100000:2.0:1789764038.334509:0:12619:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764038.334520:0:12619:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9018d0f24700 x1876699504440320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789764038.334526:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764038.334574:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764038/real 1789764038] req@ffff9018d0f24000 x1876699504439680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764038.334593:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24000 x1876699504439680/t0(0) o250->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764038.334605:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c8859000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764038.334609:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764038.334613:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8b68caea-910d-4e7e-8105-d8da387b4d8a:2389:1876699504439680:192.168.203.158@tcp:250:kworker.0 00000100:00100000:1.0:1789764038.334620:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504439808:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764038.334642:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764038.334655:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764038/real 1789764038] req@ffff9018d0f26680 x1876699504439808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764038.334669:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26680 x1876699504439808/t0(0) o38->lustre-MDT0000-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764038.334679:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764038.334684:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764038.334690:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504439808:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764038.334698:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504439936:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764038.334720:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764038.334734:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764038/real 1789764038] req@ffff9018d0f26a00 x1876699504439936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764038.334748:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f26a00 x1876699504439936/t0(0) o38->lustre-MDT0001-mdc-ffff9018c54e9800@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764038.334757:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764038.334760:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764038.334764:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f26a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504439936:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764038.334770:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504440064:192.168.203.158@tcp:8:kworker.0 00000800:00000100:1.0:1789764038.334786:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764038.334795:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764038/real 1789764038] req@ffff9018d0f25c00 x1876699504440064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764038.334806:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f25c00 x1876699504440064/t0(0) o8->lustre-OST0000-osc-ffff9018c54e9800@192.168.203.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764038.334815:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c855c000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764038.334818:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764038.334822:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f25c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:79662a0f-af46-4cf0-8a56-ded8e9c54f3e:2389:1876699504440064:192.168.203.158@tcp:8:kworker.0 00000100:00100000:1.0:1789764038.334852:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504440192:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764038.334869:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764038.334878:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764038/real 1789764038] req@ffff9018d0f27800 x1876699504440192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764038.334889:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f27800 x1876699504440192/t0(0) o38->lustre-MDT0000-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764038.334897:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c888a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764038.334900:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764038.334903:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f27800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504440192:192.168.203.158@tcp:38:kworker.0 00000100:00100000:1.0:1789764038.334908:0:2389:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504440320:192.168.203.158@tcp:38:kworker.0 00000800:00000100:1.0:1789764038.334924:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.158@tcp 00000100:00100000:1.0:1789764038.334935:0:2389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764038/real 1789764038] req@ffff9018d0f24700 x1876699504440320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764038.334946:0:2389:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9018d0f24700 x1876699504440320/t0(0) o38->lustre-MDT0001-mdc-ffff9018c5884000@192.168.203.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764038.334953:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9018c7fd9800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764038.334963:0:2389:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.158@tcp failed (-110) 00000100:00100000:1.0:1789764038.334967:0:2389:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9018d0f24700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:453b2994-2b6d-4a69-9aac-a9e10430c708:2389:1876699504440320:192.168.203.158@tcp:38:kworker.0 00010000:00010000:3.0:1789764038.974229:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 85924 lines, 85924 kept, 0 dropped, 0 bad.